Skip to main content

Introduction

Parseable is a lightweight, cloud native log observability engine. Parseable is written in Rust and uses Apache Arrow and Parquet as underlying data structures. Additionally, it uses a simple, index-free mechanism to organize and query data allowing low latency, and high throughput ingestion and query.

Parseable consumes up to ~80% lower memory and ~50% lower CPU than Elastic for similar ingestion throughput.

  • Deploy anywhere - public or private cloud, containers or VMs or bare metal environments.
  • Written in Rust, allows much smaller memory and compute foot print compared to JVM based solutions.
  • Indexing free approach, to allow high throughput ingestion, while ensuring low latency for queries.
  • Single binary with built-in UI, setup within minutes.
  • Log data stored in Parquet format, opens up opportunities for advanced analytics with vast Parquet ecosystem.

Major features

  • Choose your own storage backend - local volumes or S3 compatible object store.
  • Ingestion API, compatible with HTTP output of log agents. Refer to documentation for more details.
  • PostgreSQL based query for log search and analysis. Use Parseable UI or Grafana data source plugin for visualization.
  • Auto schema inference (schema evolution coming soon).
  • Send alerts to webhook targets including Slack.
  • Stats API to track ingestion and compressed data.

Motivation

Traditionally, logging has been seen as a text search problem. Log volumes were not high, and data ingestion or storage were not really issues. This led us to today, where all the logging platforms are primarily text search engines.

But with log data growing exponentially, today's log data challenges involve whole lot more – Data ingestion, storage, and observation, all at scale.

Parseable aims to address these challenges. Parseable is written in Rust and uses modern advancements in the data ecosystem. It uses an indexing free approach to store logs and query them - with columnar storage formats in memory (Arrow) and on disk (Parquet) to achieve high ingestion throughput and query performance.