Getting started with NiFi’s ScriptedReader by processing Key Value Pairs – Part 1

Records have become an integral part of working with NiFi since their introduction on May 8th, 2017 with the release of NiFi 1.2.0. They allow us to treat our data as more than just a bunch of bytes, giving NiFi the ability to better understand and manipulate common data formats used by other tools. However,… Continue reading Getting started with NiFi’s ScriptedReader by processing Key Value Pairs – Part 1

NiFi Terminology Basics

This post is going to cover the basic terminology that you’ll need to know for working with NiFi. Canvas The Canvas is the free grid space on which you create your Flow. It is accessed through the NiFi Web UI. You build Flows by dragging Processors on to the Canvas. Controller Service A Controller Service… Continue reading NiFi Terminology Basics

S3 Event Notifications in NiFi

The desire to pull newly uploaded files from storage is common, and typically the ListFile -> FetchFile pattern has been used in NiFi to monitor for, and then pull, new files as they arrive. Taking NiFi in to the cloud means that we’re often working with cloud object storage, like Amazon S3. While we can… Continue reading S3 Event Notifications in NiFi

Installing NiFi 1.14.0 on Linux (non-production)

This post is going to walk through installing the latest release of NiFi on a Linux host (Fedora). This latest release is version 1.14.0 and an important change is that NiFi is now secure by default. This means that HTTPS is enabled out of the box (with self signed certs), user authentication is required and… Continue reading Installing NiFi 1.14.0 on Linux (non-production)

NiFi Resources For Learning & Improving

It’s always a bit of a learning curve to get started with any new tool, not to mention keeping up to date with a tool that is under heavy active development. Here’s a list of invaluable resources to consult: NiFi Anti-Pattners by Mark Payne Mark Payne is the co-creator of NiFi and has a running… Continue reading NiFi Resources For Learning & Improving

Kafka with multiple Listeners and SASL

This will quickly discuss how to configure multiple Listeners, with the intent of having a unique Listener for External/Client traffic and another for Internal/Inter-broker traffic (and how this can be done with Cloudera Manager which requires a slight work-around in the current versions pre-2021). There’s several valid use cases for multiple Listeners. In this case,… Continue reading Kafka with multiple Listeners and SASL