Redpanda Connector

Redpanda is a Kafka compatible streaming data platform. It is JVM, ZooKeeper free and easier to operate. Redpanda is a drop-in replacement for Apache Kafka.

This document explains how to set up Redpanda, along with Connect to send logs to Parseable. We'll use Aiven's HTTP Sink Connector as a Kafka Connect plugin, and Redpanda Docker images. We assume here that you'll setup the log agent to send logs to relevant Redpanda topic.

Prerequisites

Please ensure Docker Compose installed on your machine.

Also, create a base directory and download the HTTP Sink Connector by running the following commands:

mkdir parseable
cd parseable
wget https://github.com/aiven/http-connector-for-apache-kafka/releases/download/v0.6.0/http-connector-for-apache-kafka-0.6.0.tar -O ./http-connector-for-apache-kafka-0.6.0.tar
tar -xvf ./http-connector-for-apache-kafka-0.6.0.tar -C ./

Docker Compose

Then run the following commands to start the Docker Compose. It will deploy Redpanda, Redpanda Console, Kafka Connect and Parseable.

wget https://www.parseable.com/redpanda/docker-compose.yaml
docker-compose up -d

Create the HTTP Sink Connector

Once the Docker Compose is up and running, you can create the HTTP Sink Connector. Login to the Redpanda console running at http://localhost:8080. Navigate to the Connectors tab and click on the HttpSinkConnector to create a new connector. In the next page, copy and paste the following configuration:

{
    "name": "parseable-sink",
    "connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
    "topics": "redpandademo",
    "http.url": "http://parseable:8000/api/v1/ingest",
    "http.authorization.type": "static",
    "http.headers.authorization": "Basic YWRtaW46YWRtaW4=",
    "http.headers.content.type": "application/json",
    "http.headers.additional": "X-P-Stream: redpandademo",
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
}

You should see a new connector available in the Connectors tab.

Send data to Redpanda topic

This step helps you test if everything is working as expected. In a production scenario, you'll have a log agent sending logs to the Redpanda topic.

Run the following commands to start a bash shell in the Redpanda container and post some sample data.

docker exec -it parseable-redpanda-1 bash
rpk topic produce redpandademo

The rpk command will open up a console, ready to accept events. Then paste the below data in the terminal.

{"reporterId": 8824, "reportId": 10000, "content": "Was argued independent 2002 film, The Slaughter Rule.", "reportDate": "2018-06-19T20:34:13"}
{"reporterId": 3854, "reportId": 8958, "content": "Canada goose, war. Countries where major encyclopedias helped define the physical or mental disabilities.", "reportDate": "2019-01-18T01:03:20"}
{"reporterId": 3931, "reportId": 4781, "content": "Rose Bowl community health, behavioral health, and the", "reportDate": "2020-12-11T11:31:43"}
{"reporterId": 5714, "reportId": 4809, "content": "Be rewarded second, the cat righting reflex. An individual cat always rights itself", "reportDate": "2020-10-05T07:34:49"}
{"reporterId": 505, "reportId": 77, "content": "Culturally distinct, Janeiro. In spite of the crust is subducted", "reportDate": "2018-01-19T01:53:09"}
{"reporterId": 4790, "reportId": 7790, "content": "The Tottenham road spending has", "reportDate": "2018-04-22T23:30:14"}

You can now check the Parseable dashboard at http://localhost:8000, you should see this data in the dashboard.