Parseable API calls require Basic Auth. You can find the username and password for your Parseable server in the environment variables
P_PASSWORD. If not set, the default username and password is
admin. HTTP clients generate basic auth headers from the username and password.
In case you want to manually add the basic auth header, use the following command.
echo -n '<username>:<password>' | base64
Then add the following HTTP header to the API call.
Authorization: Basic <output-from-previous-command>
Log data is ingested to Parseable via HTTP POST requests with data as JSON payload. You can use HTTP output plugins of logging agents like FluentBit, Vector, syslog-ng, LogStash among others to send log events to Parseable. You can also directly integrate Parseable with your application via REST API calls.
Batching and Compression
Max payload size in Parseable is 10 MiB (10485760 Bytes). The payload can contain single log event as a JSON object or multiple log events in a JSON array. There is no limit to number of batched events in a single call. Wherever applicable, we recommend enabling log agent's compression and batching features to reduce the network traffic and improve ingestion performance.
Log data is ingested to a Parseable log stream. Log streams are the primary unit of organization in Parseable. Log streams are identified by a unique name. In future releases, access control and retention policies will be applied at the log stream level. Currently, alerts and notifications are supported at the log stream level.
Log streams are created automatically when the first event is sent to a Parseable with a http header
x-p-stream:<stream-name>. If the stream already exists, the event is appended to the existing stream. If the stream doesn't exist, a new stream is created with the given name and the event is appended to the new stream.
Log streams have dynamic schema. This means as a user you don't have to define or modify a schema for a log stream. Parseable server detects the schema from first event and then subsequent events (with new schema) and updates internal schema accordingly. You can fetch the current schema of a log stream using the Get Schema API.
Log data formats evolve over time and users prefer a dynamic schema approach, where they don't have to worry about schema changes, and they are still able to ingest events to a given stream.
Nested JSON objects are automatically flattened. For example, the following JSON object
will be flattened to
before it gets stored. While querying, this field should be referred as
foo.bar. For example,
select foo.bar from <stream-name>. The flattened field will be available in the schema as well.
Correct time is critical in understanding the proper sequence of events. Timestamps are important for debugging, analytics, and deriving transactions. We recommend that you include a timestamp in your log events formatted in RFC3339 format.
Parseable uses the event received timestamp and adds it to the log event in the field
p_timestamp. This ensures there is a time reference in the log event, even if the original event doesn't have a timestamp.
Log stream fields are case sensitive. For example, if you send a log event like
Parseable will create two columns
Foo in the schema. So, while querying please refer to the fields as
Foo respectively. Refer to the query section below for more details.
Once the JSON payload data reaches server, it is validated and parsed to a columnar Apache Arrow format in memory. Subsequent events are appended to the Arrow record batch in memory and a copy is kept on disk (to prevent data loss). Finally, after a configurable duration, the Arrow record batch is converted to Parquet and then pushed to S3 (or compatible) bucket.
Parquet on object storage is organized into prefixes based on stream name, date and time. This ensures the server fetches specific dataset(s) based on the query time range. We're working on a compaction approach that will further compress and optimize the storage while ensuring queryable data at all times.
Parseable can use a drive (mount points) or S3 (and compatible) bucket as the backend storage. The storage mode can be configured while starting the Parseable server with the sub commands
s3-store, for drive or object storage respectively.
Based on the storage mode, server requires certain environment variables to be set. Refer the environment variables section for more details.
We recommend the
local-store mode only development and testing purposes. Note that once the server is started, the storage mode can't be changed.
To fetch the ingested data and actual compressed data size for a stream, use the Get Stats API. Sample response:
"size": "12800 Bytes"
"size": "15517 Bytes"
Parseable query API works with PostgreSQL compatible SQL. In addition to the SQL query, users need to specify the time range for which the query should be executed. The time range is specified using
endTime parameters. The response is inclusive of both the timestamps.
Check out the Query API in Postman.
Parseable uses Apache Arrow native query engine called DataFusion in conjunction with efficient Parquet reader to execute the queries.
Querying case sensitive fields
While querying, unquoted identifiers are converted to lowercase. To query column names with uppercase letters, they must be passed in double quotes. For example when sending query via the REST API, the following JSON payload will apply
WHERE condition to the column
"query":"select * from stream where \"Foo\"=bar",
If you're querying Parseable via Grafana UI (via data source plugin), you can use the following query to query the column
SELECT * FROM stream WHERE "Foo" = 'bar'