NGSI-LD: The standard interface to your IT solutions based on the Smart Data models

Step by step guide to understanding NGSI-LD and the FIWARE Context Broker for real time context data management.

With the emerging technological and digital adoption and the power of Artificial Intelligence, the use of data has become common and crucial in the tailoring of IT solutions. But the challenges surrounding data semantics, access and processing is still rising. For this reason standardisation is key.

I want to build an application which could be consumed by providing an endpoint or an interface so that the consumer could send data to it. To make this consumption scenario interoperable, it was necessary to come up with a way to standardise this endpoint/interface. This is what this article will be about.

In this article I will cover the following:

• Understanding NGSI-LD and why it is useful
• Creating an NGSI-LD data model
• Managing real time data with the FIWARE Context Broker

Motivation

Assuming you have a physical system for which you would like to create a digital twin, for this you need a data model which represents the entities that this system consists of.

You might be in a case where there is an existing data model, so first check this link of the Smart data models to find out if that is the case.

In case you found that one of the existing data models cover your use case then you should check this documentation on how to extend and reduce a data model to make it fit to your system attributes.

What is NGSI-LD and why is it useful?

NGSI-LD is a way to represent digitally a real-life object so that the machine could read it and identify it with no ambiguity.

NGSI-LD is based on JSON-LD and it introduces the concept of the @context element which provides additional information allowing the computer to interpret the rest of the data with more clarity and depth.

This enables interoperability of the solutions you could create, which is based on a standard data model with NGSI-LD. When you have a digital twin of your system and you have for each entity a respective URL/URN that uniquely identifies it.

What is the Context Broker and how it works?

The Context Broker is a building block which enables organisations to manage and share data in real-time describing “what is currently happening” within their systems and/or organisations.

The Context Broker provides the FIWARE NGSI API, which is a RESTful API enabling applications to provide updates and get access to context information.

It allows the publication of context information by entities, referred as Context Producers, that is available to other entities, referred as Context Consumers, which are interested in processing this context information. Applications or even other platform components may play the role of Context Producers, Context Consumers or both. On the other hand, updates on context information are considered as events that can be handled by applications or platform components which subscribe to those events

How to create from scratch an NGSI-LD data model

In this tutorial I will create a data model for water quality assessment. I am implementing a solution which predicts the potability of water based on water metrics which are the following: “ph”, “Hardness”, “Solids”, “Chloramines”, “Sulfate”, “Conductivity”, “Organic_carbon”, “Trihalomethane”, “Turbidity”, “Potability”.

To create a data model you need to go through the following steps:

1. Creating a baseline data model using the OpenAPI Specification
2. Validating the data model
3. Generating the NGSI-LD @context file
4. Creating entities in the Context Broker
5. Writing the API to get notifications from the Context Broker

Creating an NGSI-LD data model using the OpenAPI Specification:

To create an NGSI-LD data model you could use the Swagger online editor, once you have it save it in a <datamodel_file>.yaml

PS: the models are defined within the Smart Data Models domain.

This is a template for an NGSI-LD data model with the OpenAPI Specification:

Once you have created your data model, you can export it to a Yaml file by clicking on File>Save as Yaml

Here you can find an example of raw data_model.yaml file about a water potability system and you can inspect it with Swagger.

Validating your data model

It is necessary to check that the model is valid before processing, this can be done by viewing it online or by using a simple validator tool.

To validate you data model you need to:

1. Clone this project

$ git clone https://github.com/FIWARE/tutorials.Understanding-At-Context.git
$ cd tutorials.Understanding-At-Context
$ git checkout NGSI-LD

2. We need to activate the NGSI-LD generator tool by executing this command in the terminal:

./services create

3. Copy your <datamodel>.yaml file which you downloaded in the cloned project directory and in the terminal execute this command to validate your data model:

./services validate [datamodel.yaml]

Generating the NGSI-LD @context file

Every working linked data system relies on @context files to supply the relevant information about the data. Creating such files by hand is a tedious and error-prone procedure, so it makes sense to automate the process by executing this command in the terminal:

./services ngsi [datamodel.yaml]

By opening the generated file you can find a structure like this.

The resultant @context file is a valid JSON-LD Context file usable by NGSI-LD applications. The file is structured into three parts:

• A list of standard terms and abbreviations — this avoids the necessity of repeating URIs and reduced the overall size of the file
• A series of defined entity types (e.g. Chloramines). These usually start with a capital letter.
• A list of attributes and enumerations (not shown in this example)

Generating documentation of your data model:

The @context syntax is designed to be readable by machines. Obviously developers need human readable documentation too.

Basic documentation about NGSI-LD entities can be generated from a data model as follows:

./services markdown [datamodel.yaml]

Once you have created your model, you can upload it to GitHub and view it in a raw format.

You can share a link to the Swagger overview of your data model in this way:

https://swagger.lab.fiware.org/?url=https://raw.githubusercontent.com/<GitHub_path_to_your_data_model>/mydatamodel.yaml

This is how it should look like in Swagger:

Water potability data model overview in Swagger

In this stage, you have generated the @context file defining the context data entities which will be used in the Water Quality Assessment System. This means that you have defined an agreed set of unique IDs (URNs or URLs) for all the data entities (and every single attribute within those entities) so that other external applications will be able to programmatically understand the data held within our Context Broker.

NOTE: Another way to draft your Smart Data model is under this link

This second part of the article will cover:

• NGSI-LD Payloads — using the Accept and Content-Type Headers
• How to create NGSI-LD data entities
• Extracting the data from the Context Broker in normalized and key-value-pairs format

NGSI-LD Payloads — using the Accept and Content-Type Headers

Before starting this part, its important to clarify the following concepts:

HTTP headers can be used in an HTTP request to provide information about the request context, so that the server can tailor the response.

- the Accept headers indicate the allowed and preferred formats of the response.

- the Content-Type header is referred to as a representation header. It describes the particular representation of the resource sent in an HTTP message body.

NGSI-LD offers data in one of three formats, these effect the structure of the payload body:

• Accept: application/json - the response is in JSON format
• Accept: application/ld+json - the response is in JSON-LD format
• Accept: application/geo+json - the response is in GeoJSON or GeoJSON-LD format (not discussed in this article)

The major difference between JSON format and JSON-LD format, is that:

• If JSON-LD format is chosen, then the @context is found as an additional attribute within the body of the response.
• If the JSON only format is used the @context is passed as an additional Link Header element and is not found in the response body.

Similarly when sending NGSI-LD data to the context broker:

• An application may choose to send a payload including an additional @context attribute (in which case Content-Type: application/ld+json)
• The application may send NGSI-LD data without an additional @context attribute (in which case Content-Type: application/json and the Link header must also be present).

How to create NGSI-LD data entities

Architecture

For Demo purposes this is the used architecture, which will be deployed using Docker-Compose. It consists of:

• The Orion Context Broker which will receive requests using NGSI-LD
• The underlying MongoDB database : Used by the Orion Context Broker to hold context data information such as data entities, subscriptions and registrations
• An HTTP Web-Server which offers static @context files defining the context entities within the system. It is used to access the NGSI-LD data model on the web.

Image by the author

You can check the docker-compose.yml file used to configure the listed services under this link.

Start up

All services can be initialised from the command-line by running the services Bash script provided within this repository.

Here are the steps to create the NGSI-LD entities:

1. clone this project

$ git clone https://github.com/FIWARE/tutorials.Getting-Started.git 
$ cd tutorials.Getting-Started 
$ git checkout NGSI-LD

2. Activate the services by running this command in the terminal:

./services start

You have learned already how to generate the @context file ngsi-context.jsonld which serves to define all attributes when sending data to the Context Broker or retrieving data in normalized format. This @context must be used for all NGSI-LD to NGSI-LD interactions. Now it is time to create the entities following these steps:

3. Copy your generated data model @context file ngsi-context.jsonld under tutorials.Getting-Started/data-models

You should be able to access your data model on the web on port 3004 of the httpd server: http://localhost:3004/<your-mgsi-context>.jsonld

Image by the author

New context data entities can be created by making a POST request to the /ngsi-ld/v1/entities endpoint and supply an @context along with structured NGSI-LD data.

4. In your terminal use the following command to create a pH entity:

curl -iX POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/json' \
-H 'Link: <http://context/water-ngsi.jsonld>;  rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d '{
    "id": "urn:ngsi-ld:WaterPotabilityMetrics:001",
    "type": "WaterPotabilityMetrics",
    "pH": {
        "type": "Property",
        "value": 6
    }}'

Make sure to adapt this command to your needs and especially:

• The link header of the http request in which you need to pass your @context file. ( e.g -H ‘Link: <http://context/water-ngsi.jsonld>; rel=”http://www.w3.org/ns/json-ld#context"; type=”application/ld+json”)
• The payload in which you precise which entities you create, id and type corresponding to your data model.

Extracting the data from the Context Broker in normalized and key-value-pairs format.

The two main data formats are normalized and key-value-pairs.

• Data returned in the normalised format respects the NGSI-LD rules and may be used directly by another Context Broker (or any other component offering an NGSI-LD interface).
• Data returned in the key-value-pairs format is by definition not NGSI-LD, But it could be used in applications where its target is to get the value of the entities/attributes.

To get the desired data format, all you have to do is precise the option header in your HTTP request when getting data from the Context Broker.

For normalised data format:

For KeyValue data format:

In addition to the Accept and Link header, it is necessary to add to the Query Parameters: options=keyValues

To get all the entities with type=WaterPotabilityMetrics use this command:

curl -G -X GET \
    'http://localhost:1026/ngsi-ld/v1/entities' \
-H 'Link: <http://context/water-ngsi.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-H 'Accept: application/json' \
-d 'options=keyValues' \
-d 'type=WaterPotabilityMetrics'

Check this link to find out more about the queries option.

Prerequisites

Swagger

The OpenAPI Specification (commonly known as Swagger) is an API description format for REST APIs. A Swagger spec allows you to describe an entire API (such as NGSI-LD itself) however in this tutorial you shall be concentrating on using Swagger to define data models.

API specifications can be written in YAML or JSON. The format is easy to learn and readable to both humans and machines. The complete OpenAPI Specification can be found on GitHub: OpenAPI 3.0 Specification.

This is important since you will need a well defined structure to be able to generate @context files.

Docker

To keep things interoperable across different environments, the file generator is running on Docker. Docker is a container technology which allows to different components isolated into their respective environments.

• To install Docker on Windows follow the instructions here
• To install Docker on Mac follow the instructions here
• To install Docker on Linux follow the instructions here

Check this article to understand Docker and containerisation concepts 👇

All you need to know about Docker, Docker-Compose and Dockerfile.

Let’s wrap it up. This post was an opportunity to:

• Understand what NGSI-LD is.
• Why it is useful to have standard data models.
• Manage context data with the FIWARE Context Broker.
• How to standardise the data access and management in IT solutions.

I hope this article was insightful and stay tuned for more articles ;)

Leave me your comments if you have any questions.