Enriching Records with LookupRecord & REST APIs in NiFi

This is a pattern I have seen quite frequently, especially in IoT flows. At a high level: we have a stream of sensor data coming in from our IoT devices, and an external service that contains additional contextual data exposed via a REST API. With every sensor message, we want to call the REST API,… Continue reading Enriching Records with LookupRecord & REST APIs in NiFi

Kafka with multiple Listeners and SASL

This will quickly discuss how to configure multiple Listeners, with the intent of having a unique Listener for External/Client traffic and another for Internal/Inter-broker traffic (and how this can be done with Cloudera Manager which requires a slight work-around in the current versions pre-2021). There’s several valid use cases for multiple Listeners. In this case,… Continue reading Kafka with multiple Listeners and SASL

Moving the data dir of MariaDB on CentOS7/RHEL7

You should have root access to the CentOS host and a new target directory ready for the MariaDB data. For this guide, our new target directory is /data/database. First, stop MariaDB. systemctl stop mariadb Now, copy your existing database directory to the new location. By default, it is /var/lib/mysql. If it’s not there, check the… Continue reading Moving the data dir of MariaDB on CentOS7/RHEL7

Modern Streaming Architectures – Intro

It’s 2020 and we’re creating more data than ever. We each use the internet, in some way or another, for almost eveything we do. Listening to music, reading a book, playing a game, managing our money, talking to our friends and family – these are all tasks that used to be offline experiences that we’ve… Continue reading Modern Streaming Architectures – Intro