Kafka Connector

Log agents like FluentBit, Vector etc run as standalone services on host machines, forwarding logs as they are generated. In some cases, this approach might be insufficient. Users may expect certain guarantees in log delivery, buffering and high availability.

Kafka comes in handy in such cases. Kafka sits between log agents and log search/query platforms. Enterprises use Kafka as a central processing layer to hold the logs temporarily to handle spikes and unexpected storage failures.

This document explains how to set up Kafka, along with Kafka Connect to send logs to Parseable. We'll use Aiven's HTTP Sink Connector as a Kafka Connect plugin, and Confluent Docker images for Kafka distribution. We assume here that you'll setup the log agent to send logs to relevant Kafka topic.

Docker Compose

Please ensure Docker Compose installed on your machine.

Then run the following commands to start the Docker Compose. It will deploy Kafka, Kafka Connect, Zookeeper and Parseable.

mkdir parseable
cd parseable
wget https://www.parseable.io/kafka/docker-compose.yaml
docker-compose up -d

Sending data to Kafka topic

This step helps you test if everything is working as expected. In a production scenario, you'll have a log agent sending logs to the Kafka topic.

We'll send data to the demo Kafka topic already created in the Docker Compose. Connector plugin would then automatically send this data to Parseable, and it should be visible in the Parseable UI. Exec into the Kafka container and list the topics. You should see a topic called kafkademo.

docker exec -it kafka-kafka-1 /bin/bash
kafka-topics --list --bootstrap-server localhost:9092

Then use the Kafka console producer to send some demo data to the topic. From within the Kafka container, run the following command.

kafka-console-producer --bootstrap-server localhost:9092 --topic kafkademo

You should see a prompt > to enter the data. Enter the following JSON objects one by one.

{"reporterId": 8824, "reportId": 10000, "content": "Was argued independent 2002 film, The Slaughter Rule.", "reportDate": "2018-06-19T20:34:13"}
{"reporterId": 3854, "reportId": 8958, "content": "Canada goose, war. Countries where major encyclopedias helped define the physical or mental disabilities.", "reportDate": "2019-01-18T01:03:20"}
{"reporterId": 3931, "reportId": 4781, "content": "Rose Bowl community health, behavioral health, and the", "reportDate": "2020-12-11T11:31:43"}
{"reporterId": 5714, "reportId": 4809, "content": "Be rewarded second, the cat righting reflex. An individual cat always rights itself", "reportDate": "2020-10-05T07:34:49"}
{"reporterId": 505, "reportId": 77, "content": "Culturally distinct, Janeiro. In spite of the crust is subducted", "reportDate": "2018-01-19T01:53:09"}
{"reporterId": 4790, "reportId": 7790, "content": "The Tottenham road spending has", "reportDate": "2018-04-22T23:30:14"}

Now, to check if Parseable received the data, login to the Parseable UI on http://localhost:8000. Then select the kafkademo stream from the dropdown. You should see the data in the UI.

Docker Compose​

Sending data to Kafka topic​

Docker Compose

Sending data to Kafka topic