Elixir Broadway Auth Plugin for Amazon MSK

Background

Elixir is a modern programming language designed for concurrency and distributed computing. It provides developers with powerful abstractions for building concurrent applications.GenStageis one such Elixirbehaviourthat allows exchanging events with back-pressure between Elixir processes. Broadway is a data processing and ingestion library for Elixir, built on top of GenStage behavior. It allows developers to define a series of stages for data processing. Each stage processes data in parallel and can be configured to handle back-pressure, ensuring that data is processed efficiently without overloading downstream systems. Broadway enables developers to build robust, concurrent, scalable, and resilient data processing pipelines with minimal effort. Broadway allows applications to consume data from various sources (aka producers) like Amazon SQS, Apache Kafka, Google Cloud PubSub, RabbitMQ, etc.,

Problem

Broadway-Kafkais a Broadway source connector for Apache Kafka - in simple terms, it combines features of Broadway with a Kafka client consumer. Amazon MSK (Managed Streaming for Apache Kafka) is a leading provider of Kafka as a Managed Service. As of Q1 2022, AWS accounts for33% of global cloud infrastructure, which means MSK would be a top choice for customers evaluating Kafka cloud deployment. Amazon MSK supports multiple auth mechanisms andIAMis one of them. The main advantage of using IAM for MSK in an AWS environment is that it helps manage all roles and permissions for a company’s application suite in a single place. As of today, there’s no support for Broadway to talk with Amazon MSK using IAM authentication.

Solution

MSK Supporting IAM

To bring IAM support for MSK, AWS has made minor modifications to Apache Kafka’s source code. IAM authentication is offered as a custom Simple Authentication and Security Layer (SASL) mechanism named AWS_MSK_IAM. This means only clients that can authenticate via the custom SASL mechanism can produce/consume messages with IAM protected MSK.

Broadway-Kafka

Broadway-Kafka gets Kafka consumer capabilities by relying onBrod, an Erlang library to work with Apache Kafka for Erlang/Elixir: Brod supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 authentication mechanisms out of the box. It also provides a behavior brod_auth_backend that acts as an extension point for custom authentication plugins.

We created ex_aws_msk_iam_auth, an Elixir plugin library to authenticate with MSK via the AWS_MSK_IAM SASL mechanism. Since Elixir and Erlang are interoperable, this library implements the brod_auth_backend behavior. The plugin is configurable as part of Broadway’s producer client configuration. As part of the authentication handshake, the plugin generates a payload and signs it, using an AWS4 signature.

For more details and usage, please visit itsGitHub repository

Aside

While implementing this library, we noticed that although Brod supports authentication extensions, Broadway-Kafka abstracted it out. Here’s ourpull requestaddressing the issue to support custom Brod SASL authenticators and enabled it in Broadway-Kafka,

Summary

The Elixir Broadway Auth Plugin for Amazon MSK empowers developers to build secure and scalable event-streaming applications using Elixir and Amazon MSK. By providing robust authentication and fine-grained authorization capabilities, the plugin simplifies the process of securing your Broadway pipelines while ensuring only authorized actors can access and interact with your Kafka cluster. With the power of Elixir Broadway and the added security of the Auth Plugin, developers can confidently leverage the full potential of event-driven architectures in their applications.