Fluent Bit is a lightweight, and scalable logging and metrics processor and forwarder. It is one of the most popular choices in cloud and container environments. Fluent Bit agents can be configured to send logs to Parseable with HTTP output plugin and JSON output format.
This document explains how to set up Fluent Bit to ship logs to Parseable. We use Fluent Bit's Memory Metrics Input plugin as the source of logs and send the data to Parseable using the HTTP output plugin.
In this document we'll cover a simple Docker Compose based setup and a Kubernetes based deployment with Helm.
Please ensure Docker Compose installed on your machine. Then run the following commands to set up Parseable and Fluent Bit.
docker-compose up -d
You can now access the Parseable dashboard on http://localhost:8000. You should see a log stream called
fluentbitdemo populated with log data generated by the Memory Metrics Input plugin.
- Please ensure
helminstalled and configured to access your Kubernetes cluster.
- Parseable installed on your Kubernetes cluster. Refer the Parseable Kubernetes documentation here: https://www.parseable.io/docs/installation/kubernetes.
Install Fluent Bit
We use the official Fluent Bit Helm chart. But, we'll use a modified
values.yaml file, that contains the configuration for Fluent Bit to send logs to Parseable.
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit --values values.yaml -n fluentbit --create-namespace
Let's take a deeper look at the Fluent Bit configuration in
values.yaml. Here we use the
kubernetes filter to enrich the logs with Kubernetes metadata. We then use the
http output plugin to send logs to Parseable. Notice the
Match section in the
http output plugin. We use
kube.* to match all logs from Kubernetes filter. With the header
X-P-Stream fluentbitdemo, we tell Parseable to send the logs to the
header Content-Type application/json
header X-P-META-meta1 value1
header X-P-TAG-tag1 value1
header X-P-Stream fluentbitdemo
Batching and Compression
Parseable supports batching and compressing the log data before sending it via HTTP POST. Fluent Bit supports this feature via the
buffer_max_size option. We recommend enabling both of these options to reduce the number of HTTP requests and to reduce the size of the HTTP payload.
Check logs in Parseable
If you've not already done so, port-forward Parseable service to access the dashboard with:
kubectl port-forward svc/parseable 8000:80 -n parseable
You can now check the Parseable server
fluentbitdemo stream to see the logs from this setup.