Sdairs

Category: Tech

  • S3 Event Notifications in NiFi

    The desire to pull newly uploaded files from storage is common, and typically the ListFile -> FetchFile pattern has been used in NiFi to monitor for, and then pull, new files as they arrive. Taking NiFi in to the cloud means that we’re often working with cloud object storage, like Amazon S3. While we can […]

  • Installing NiFi 1.14.0 on Linux (non-production)

    This post is going to walk through installing the latest release of NiFi on a Linux host (Fedora). This latest release is version 1.14.0 and an important change is that NiFi is now secure by default. This means that HTTPS is enabled out of the box (with self signed certs), user authentication is required and […]

  • Importing individual flow definitions in NiFi

    While I recommend you Version Control & Deploy flows with NiFi Registry you can also quickly import flow definition json files straight from the NiFi GUI. This post will quickly demonstrate how. To get some json files to work with, clone my nifi-snippets repo that contains all of the NiFi flow demos that I have […]

  • Building a simple REST API in NiFi

    In a previous post I discussed using REST APIs to enrich records at the time of ingest. This post will cover building the corresponding REST API that I used in that post. This will be a very simple REST API that exposes a single endpoint GET /api. This endpoint returns a 200 OK response with […]

  • Enriching Records with LookupRecord & REST APIs in NiFi

    This is a pattern I have seen quite frequently, especially in IoT flows. At a high level: we have a stream of sensor data coming in from our IoT devices, and an external service that contains additional contextual data exposed via a REST API. With every sensor message, we want to call the REST API, […]

  • NiFi in Production – Version Control & Deploy flows with NiFi Registry

    This post describes and demonstrates how to use NiFi and NiFi registry to develop flows in Dev, version control the flow, and then deploy the versioned flow to Prod.

  • NiFi Resources For Learning & Improving

    It’s always a bit of a learning curve to get started with any new tool, not to mention keeping up to date with a tool that is under heavy active development. Here’s a list of invaluable resources to consult: NiFi Anti-Pattners by Mark Payne Mark Payne is the co-creator of NiFi and has a running […]

  • Kafka with multiple Listeners and SASL

    This will quickly discuss how to configure multiple Listeners, with the intent of having a unique Listener for External/Client traffic and another for Internal/Inter-broker traffic (and how this can be done with Cloudera Manager which requires a slight work-around in the current versions pre-2021). There’s several valid use cases for multiple Listeners. In this case, […]

  • Connecting to a secure Impala via NiFi with TLS and Kerberos (and Hive2)

    There’s a few different ways this could be done. I’ll demonstrate one possible way, using ExecuteSQL to connect to Impala via the JDBC driver.

  • Moving the data dir of MariaDB on CentOS7/RHEL7

    You should have root access to the CentOS host and a new target directory ready for the MariaDB data. For this guide, our new target directory is /data/database. First, stop MariaDB. systemctl stop mariadb Now, copy your existing database directory to the new location. By default, it is /var/lib/mysql. If it’s not there, check the […]