Converting JSON to CSV with Apache NiFi

Apache NiFi is an open source software for automating and managing the data flow between systems. It provides web-based User Interface to create, monitor, and control data flows. Its main components are FlowFiles, which represent each piece of data and Processors, responsible for creating, sending, receiving, transforming, routing, splitting, merging, and processing FlowFiles.

This article covers the steps to perform the conversion of files in JSON format to CSV using ConvertRecord processor. There are many other ways to perform Data transformation from JSON to CSV but this is the most straight forward and simple way.

Let’s get into it!!

This is the general overview of the proposed solution:

I will go through each processor and explain the properties to be configured.

GetFile processor

This processor creates FlowFiles from files in a directory. We will use GetFile to read the data which will be initially in JSON format. All we need to configure is the Input Directory which represents the path to our files.

ConvertRecord processor

ConvertRecord converts records from one data format to another using configured Record Reader and Record Write Controller Services. The Reader and Writer must be configured with “matching” schemas. By this, we mean the schemas must have the same field names.

Since we want to convert JSON to CSV then we will need to use JsonTreeReader as a Record Reader and CSVRecordSetWriter

The next step is to configure JsonTreeReader and CSVRecordSetWriter and what we need to do is to create an AvroSchemaRegistry which represents your data in JSON format in Avro format.

To create and verify your Avro schema, you can use the following website.

JsonTreeReader

AvroSchemaRegistry

CsvRecordSetWriter

This is all you need to make the conversion from JSON to CSV.

Input

Output

I hope this article was helpful.

By Rihab Feki

Machine Learning engineer and Evangelist at FIWARE.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store