Hands on Apache NiFi: Converting JSON to CSV

Picture by the author

Apache NiFi is an open source software for automating and managing the data flow between systems. It provides web-based User Interface to create, monitor, and control data flows. Its main components are FlowFiles, which represent each piece of data and Processors, responsible for creating, sending, receiving, transforming, routing, splitting, merging, and processing FlowFiles.

This article covers the steps to perform the conversion of files in JSON format to CSV using ConvertRecord processor. There are many other ways to perform Data transformation from JSON to CSV but this is the most straight forward and simple way.

Let’s get into it!!

This is the general overview of the proposed solution:

Picture by the author

I will go through each processor and explain the properties to be configured.

GetFile processor

This processor creates FlowFiles from files in a directory. We will use GetFile to read the data which will be initially in JSON format. All we need to configure is the Input Directory which represents the path to our files.

Picture by the author

ConvertRecord processor

ConvertRecord converts records from one data format to another using configured Record Reader and Record Write Controller Services. The Reader and Writer must be configured with “matching” schemas. By this, we mean the schemas must have the same field names.

Since we want to convert JSON to CSV then we will need to use JsonTreeReader as a Record Reader and CSVRecordSetWriter

Picture by the author

The next step is to configure JsonTreeReader and CSVRecordSetWriter and what we need to do is to create an AvroSchemaRegistry which represents your data in JSON format in Avro format.

To create and verify your Avro schema, you can use the following website.

JsonTreeReader

Picture by the author

AvroSchemaRegistry

Picture by the author

CsvRecordSetWriter

Picture by the author

This is all you need to make the conversion from JSON to CSV.

Input

Picture by the author

Output

Picture by the author

I hope this article was helpful.

By Rihab Feki

Machine Learning engineer and Evangelist at FIWARE.