-
The Cloudera Innovation Accelerator & me
I recently took on a new role at Cloudera as a Senior Developer Advocate in the newly formed Cloudera Innocation Accelerator. This exciting new team sits between Cloudera and the wider data community, looking at new & ermeging technologies that are in-demand & have real potential to provide value to Cloudera’s customers. In this role, […]
-
Introducing dbt-impala
As part of my new role at Cloudera, I have been looking at a tool called dbt. To many, dbt needs no introduction – it has not just made a splash in the data ecosystem, it has come in riding a tsunami of positivity, and for good reason! Just looking at it’s star history we […]
-
Getting started with NiFi’s ScriptedReader by processing Key Value Pairs – Part 1
Records have become an integral part of working with NiFi since their introduction on May 8th, 2017 with the release of NiFi 1.2.0. They allow us to treat our data as more than just a bunch of bytes, giving NiFi the ability to better understand and manipulate common data formats used by other tools. However, […]
-
NiFi Terminology Basics
This post is going to cover the basic terminology that you’ll need to know for working with NiFi. Canvas The Canvas is the free grid space on which you create your Flow. It is accessed through the NiFi Web UI. You build Flows by dragging Processors on to the Canvas. Controller Service A Controller Service […]
-
S3 Event Notifications in NiFi
The desire to pull newly uploaded files from storage is common, and typically the ListFile -> FetchFile pattern has been used in NiFi to monitor for, and then pull, new files as they arrive. Taking NiFi in to the cloud means that we’re often working with cloud object storage, like Amazon S3. While we can […]
-
Installing NiFi 1.14.0 on Linux (non-production)
This post is going to walk through installing the latest release of NiFi on a Linux host (Fedora). This latest release is version 1.14.0 and an important change is that NiFi is now secure by default. This means that HTTPS is enabled out of the box (with self signed certs), user authentication is required and […]
-
Importing individual flow definitions in NiFi
While I recommend you Version Control & Deploy flows with NiFi Registry you can also quickly import flow definition json files straight from the NiFi GUI. This post will quickly demonstrate how. To get some json files to work with, clone my nifi-snippets repo that contains all of the NiFi flow demos that I have […]
-
Building a simple REST API in NiFi
In a previous post I discussed using REST APIs to enrich records at the time of ingest. This post will cover building the corresponding REST API that I used in that post. This will be a very simple REST API that exposes a single endpoint GET /api. This endpoint returns a 200 OK response with […]
-
Enriching Records with LookupRecord & REST APIs in NiFi
This is a pattern I have seen quite frequently, especially in IoT flows. At a high level: we have a stream of sensor data coming in from our IoT devices, and an external service that contains additional contextual data exposed via a REST API. With every sensor message, we want to call the REST API, […]
-
NiFi in Production – Version Control & Deploy flows with NiFi Registry
This post describes and demonstrates how to use NiFi and NiFi registry to develop flows in Dev, version control the flow, and then deploy the versioned flow to Prod.