Vector

Vector is a lightweight, ultra-fast tool for building observability pipelines. Vector can be configured to send logs to Parseable with HTTP Sink and JSON codec.

This document explains how to set up Vector to ship logs to Parseable on a Kubernetes cluster. We use Vector's kubernetes_logs source to collect logs from Kubernetes pods. Then we use the http sink to send logs to Parseable.

Kubernetes

• Please ensure kubectl and helm installed and configured to access your Kubernetes cluster.
• Parseable installed on your Kubernetes cluster. Refer the Parseable Kubernetes documentation here: https://www.parseable.com/docs/installation/kubernetes-helm.

Install Vector

We use the official Vector Helm chart. But, we'll use a modified values.yaml file, that contains the configuration for Fluent Bit to send logs to Parseable.

wget https://www.parseable.com/vector/values.yaml
helm repo add vector https://helm.vector.dev

helm install vector vector/vector \
  --namespace vector \
  --create-namespace \
  --values values.yaml

Let's take a deeper look at the Vector sink configuration in values.yaml. Important to notice here is that we use the sink type http. Source can be any of the supported sources by Vector.

  sources:
    kubernetes_logs:
      type: kubernetes_logs
  sinks:
    parseable:
      type: http
      method: post
      batch:
        max_bytes: 10485760
        max_events: 1000
        timeout_secs: 10
      compression: gzip
      inputs: 
        - kubernetes_logs
      encoding:
        codec: json
      uri: 'http://parseable.parseable.svc.cluster.local/api/v1/ingest'
      auth:
        strategy: basic
        user: admin
        password: admin
      request:
        headers:
          X-P-Stream: vectordemo
      healthcheck:
        enabled: true
        path: 'http://parseable.parseable.svc.cluster.local/api/v1/liveness'
        port: 80

Batching and Compression

Parseable supports batching and compressing the log data before sending it via HTTP POST. Vector supports this feature via the batch and compression options. We recommend enabling both of these options to reduce the number of HTTP requests and to reduce the size of the HTTP payload.

Check logs in Parseable

If you've not already done so, port-forward Parseable service to access the dashboard with:

kubectl port-forward svc/parseable 8000:80 -n parseable

You can now check the Parseable server vectordemo stream to see the logs from this setup.