Enrich 6.11.x upgrade guide with compression
Enrich 6.11.0 introduces enriched stream compression, a feature that reduces the size (and therefore, cost) of data written to your enriched output stream.
Enrich compresses enriched events before writing them to the output stream. This provides several benefits:
- Reduced storage costs — compressed events take up less space in your output streams
- Improved throughput — smaller records reduce the overhead of stream processing
- Faster downstream processing — consumers handle smaller messages more efficiently
Enabling compression
Before enabling compression, you must ensure your downstream consumers support decompression. The following applications are compatible:
- BigQuery Loader (since 2.2.0)
- Databricks Streaming Loader (since 0.5.0)
- Elasticsearch Loader (since 3.0.0)
- Lake Loader (since 0.10.0)
- S3 Loader (since 3.2.0)
- Snowflake Loader (since 0.6.0)
- Snowbridge (since 5.2.0)
Do not enable this setting if your downstream consumer is not in this list.
When upgrading to Enrich 6.11.0, compression is an optional feature that you can configure in your Enrich settings. If you don't enable this feature, the data format and size remain unchanged.
After upgrading your downstream consumers, enable compression in Enrich by adding the following config section:
compression {
enabled = true
}
See the configuration reference for additional options such as the compression algorithm (zstd or gzip) and compression level.
Impact on metrics
When compression is enabled, there will be a decrease in the size of messages sent to the enriched event stream, i.e. Kinesis, Pub/Sub or Kafka, depending on your cloud. You will notice this decrease if you monitor metrics on the enriched stream.
This is perfectly normal and does not indicate any drop in event volumes.