Enrich 6.11.x upgrade guide with compression
Enrich 6.11.0 introduces enriched stream compression, a feature that reduces the size (and therefore, cost) of data written to your enriched output stream.
Enrich compresses enriched events before writing them to the output stream. This provides several benefits:
- Reduced storage costs — compressed events take up less space in your output streams
- Improved throughput — smaller records reduce the overhead of stream processing
- Faster downstream processing — consumers handle smaller messages more efficiently
Enabling compression
Every application that reads from your enriched stream must support compressed records before you enable compression. All Snowplow applications support them from these versions:
- BigQuery Loader (since 2.2.0)
- Databricks Streaming Loader (since 0.5.0)
- Elasticsearch Loader (since 3.0.0)
- Lake Loader (since 0.10.0)
- S3 Loader (since 3.2.0)
- Snowflake Loader (since 0.6.0)
- Snowbridge (since 5.2.0)
If you run your own consumer of the enriched stream, update it to read compressed records first.
When upgrading to Enrich 6.11.0, compression is an optional feature that you can configure in your Enrich settings. If you don't enable this feature, the data format and size remain unchanged.
After upgrading your downstream consumers, enable compression in Enrich by adding the following config section:
compression {
enabled = true
}
See the configuration reference for additional options such as the compression algorithm (zstd or gzip) and compression level.
When compression is enabled, Enrich batches events with different field values into one stream record, so it does not apply output.good.partitionKey (Kinesis and Kafka) or output.good.attributes (Pub/Sub message attributes and Kafka record headers) to the enriched stream. If a downstream consumer relies on these, change it to read the values from the enriched events instead.
Impact on metrics
When compression is enabled, there will be a decrease in the size of messages sent to the enriched event stream, i.e. Kinesis, Pub/Sub or Kafka, depending on your cloud. You will notice this decrease if you monitor metrics on the enriched stream.
This is perfectly normal and does not indicate any drop in event volumes.