Enriching Records with LookupRecord & REST APIs in NiFi

This is a pattern I have seen quite frequently, especially in IoT flows. At a high level: we have a stream of sensor data coming in from our IoT devices, and an external service that contains additional contextual data exposed via a REST API. With every sensor message, we want to call the REST API,… Continue reading Enriching Records with LookupRecord & REST APIs in NiFi

Kafka with multiple Listeners and SASL

This will quickly discuss how to configure multiple Listeners, with the intent of having a unique Listener for External/Client traffic and another for Internal/Inter-broker traffic (and how this can be done with Cloudera Manager which requires a slight work-around in the current versions pre-2021). There’s several valid use cases for multiple Listeners. In this case,… Continue reading Kafka with multiple Listeners and SASL

Modern Streaming Architectures – Ingest

If you didn’t catch the previous post, you can check it out here: Modern Streaming Architectures – From The Sky. Ingest sounds simple, but can be hard to get right. We often have many different sources of data, sending in different formats, in different volumes, with different schedules, different delivery guarentees and different delivery mechanisms.… Continue reading Modern Streaming Architectures – Ingest

Moving the data dir of MariaDB on CentOS7/RHEL7

You should have root access to the CentOS host and a new target directory ready for the MariaDB data. For this guide, our new target directory is /data/database. First, stop MariaDB. systemctl stop mariadb Now, copy your existing database directory to the new location. By default, it is /var/lib/mysql. If it’s not there, check the… Continue reading Moving the data dir of MariaDB on CentOS7/RHEL7

Modern Streaming Architectures – Intro

It’s 2020 and we’re creating more data than ever. We each use the internet, in some way or another, for almost eveything we do. Listening to music, reading a book, playing a game, managing our money, talking to our friends and family – these are all tasks that used to be offline experiences that we’ve… Continue reading Modern Streaming Architectures – Intro