Hands on Apache NiFi: Converting JSON to CSV
Apache NiFi is an open source software for automating and managing the data flow between systems. It provides web-based User Interface to create, monitor, and control data flows. Its main components are FlowFiles, which represent each piece of data and Processors, responsible for creating, sending, receiving, transforming, routing, splitting, merging, and processing FlowFiles.
This article covers the steps to perform the conversion of files in JSON format to CSV using ConvertRecord processor. There are many other ways to perform Data transformation from JSON to CSV but this is the most straight forward and simple way.
Let’s get into it!!
This is the general overview of the proposed solution:
I will go through each processor and explain the properties to be configured.
GetFile processor
This processor creates FlowFiles from files in a directory. We will use GetFile to read the data which will be initially in JSON format. All we need to configure is the Input Directory which represents the path to our files.
ConvertRecord processor
ConvertRecord converts records from one data format to another using configured Record Reader and Record Write Controller Services. The Reader and Writer must be configured with “matching” schemas. By this, we mean the schemas must have the same field names.
Since we want to convert JSON to CSV then we will need to use JsonTreeReader as a Record Reader and CSVRecordSetWriter
The next step is to configure JsonTreeReader and CSVRecordSetWriter and what we need to do is to create an AvroSchemaRegistry which represents your data in JSON format in Avro format.
To create and verify your Avro schema, you can use the following website.
JsonTreeReader
AvroSchemaRegistry
CsvRecordSetWriter
This is all you need to make the conversion from JSON to CSV.
Input
Output
I hope this article was helpful.
By Rihab Feki
