Getting started with NiFi’s ScriptedReader by processing Key Value Pairs – Part 1

Records have become an integral part of working with NiFi since their introduction on May 8th, 2017 with the release of NiFi 1.2.0. They allow us to treat our data as more than just a bunch of bytes, giving NiFi the ability to better understand and manipulate common data formats used by other tools. However,… Continue reading Getting started with NiFi’s ScriptedReader by processing Key Value Pairs – Part 1

S3 Event Notifications in NiFi

The desire to pull newly uploaded files from storage is common, and typically the ListFile -> FetchFile pattern has been used in NiFi to monitor for, and then pull, new files as they arrive. Taking NiFi in to the cloud means that we’re often working with cloud object storage, like Amazon S3. While we can… Continue reading S3 Event Notifications in NiFi

Enriching Records with LookupRecord & REST APIs in NiFi

This is a pattern I have seen quite frequently, especially in IoT flows. At a high level: we have a stream of sensor data coming in from our IoT devices, and an external service that contains additional contextual data exposed via a REST API. With every sensor message, we want to call the REST API,… Continue reading Enriching Records with LookupRecord & REST APIs in NiFi