Transformer Kinesis configuration reference

The configuration reference in this page is written for Transformer Kinesis 6.0.0

An example of the minimal required config for the Transformer Kinesis can be found here and a more detailed one here.

License

Since version 6.0.0, RDB Loader is released under the Snowplow Limited Use License (FAQ).

To accept the terms of license and run RDB Loader, set the ACCEPT_LIMITED_USE_LICENSE=yes environment variable. Alternatively, you can configure the license.accept option, like this:

license {
  accept = true
}

Parameter	Description
`input.appName`	Optional. KCL app name. The default is `snowplow-rdb-transformer`
`input.streamName`	Required for `kinesis`. Enriched Kinesis stream name.
`input.region`	AWS region of the Kinesis stream. Optional if it can be resolved with AWS region provider chain.
`input.position`	Optional. Kinesis position: `LATEST` or `TRIM_HORIZON`. The default is `LATEST`.
`windowing`	Optional. Frequency to emit shredding complete message. The default is `10 minutes`. Maximum allowed value is `60 minutes`
`output.path`	Required. S3 URI of the transformed output.
`output.compression`	Optional. One of `NONE` or `GZIP`. The default is `GZIP`.
`output.region`	AWS region of the S3 bucket. Optional if it can be resolved with AWS region provider chain.
`output.maxRecordsPerFile` (since 5.4.0)	Optional. Default = 10000. Max number of events per parquet partition.
`output.bad.type` (since 5.4.0)	Optional. Either `kinesis` or `file`, default value `file`. Type of bad output sink. When `file`, badrows are written as files under URI configured in `output.path`.
`output.bad.streamName` (since 5.4.0)	Required if output type is `kinesis`. Name of the Kinesis stream to write to.
`output.bad.region` (since 5.4.0)	AWS region of the Kinesis stream. Optional if it can be resolved with AWS region provider chain.
`output.bad.recordLimit` (since 5.4.0)	Optional, default = 500. Limits the number of events in a single PutRecords Kinesis request.
`output.bad.byteLimit` (since 5.4.0)	Optional, default = 5242880. Limits the number of bytes in a single PutRecords Kinesis request.
`output.bad.backoffPolicy.minBackoff` (since 5.4.0)	Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails with internal errors.
`output.bad.backoffPolicy.maxBackoff` (since 5.4.0)	Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails with internal errors.
`output.bad.backoffPolicy.maxRetries` (since 5.4.0)	Optional, default = 10. Maximum number of retries for internal Kinesis errors.
`output.bad.throttledBackoffPolicy.minBackoff` (since 5.4.0)	Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails in case of throughput exceeded.
`output.bad.throttledBackoffPolicy.maxBackoff` (since 5.4.0)	Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails in case of throughput exceeded. Writing is retried forever.
`queue.type`	Required. Type of the message queue. Can be either `sqs` or `sns`.
`queue.queueName`	Required if queue type is `sqs`. Name of the SQS queue. SQS queue needs to be FIFO.
`queue.topicArn`	Required if queue type is `sns`. ARN of the SNS topic.
`queue.region`	AWS region of the SQS queue or SNS topic. Optional if it can be resolved with AWS region provider chain.
`formats.*`	Schema-specific format settings.
`formats.transformationType`	Required. Type of transformation, either `shred` or `widerow`. See Shredded data and Wide row format.
`formats.fileFormat`	Optional. The default is `JSON`. Output file format produced when transformation is `widerow`. Either `JSON` or `PARQUET`.
`formats.default`	Optional. The default is `TSV`. Data format produced by default when transformation is `shred`. Either `TSV` or `JSON`. `TSV` is recommended as it enables table autocreation, but requires an Iglu Server to be available with known schemas (including Snowplow schemas). `JSON` does not require an Iglu Server, but requires Redshift JSONPaths to be configured and does not support table autocreation.
`formats.tsv`	Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `JSON` this list of schemas will still be shredded into `TSV`.
`formats.json`	Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `TSV` this list of schemas will still be shredded into `JSON`.
`formats.skip`	Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. Schemas for which loading can be skipped.
`monitoring.metrics.*`	Send metrics to a StatsD server or stdout.
`monitoring.metrics.statsd.*`	Optional. For sending metrics (good and bad event counts) to a StatsD server.
`monitoring.metrics.statsd.hostname`	Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.
`monitoring.metrics.statsd.port`	Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.
`monitoring.metrics.statsd.tags`	Optional. Tags which are used to annotate the StatsD metric with any contextual information.
`monitoring.metrics.statsd.prefix`	Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.
`monitoring.metrics.stdout.*`	Optional. For sending metrics to stdout.
`monitoring.metrics.stdout.prefix`	Optional. Overrides the default metric prefix.
`telemetry.disable`	Optional. Set to `true` to disable telemetry.
`telemetry.userProvidedId`	Optional. See here for more information.
`monitoring.sentry.dsn`	Optional. For tracking runtime exceptions.
`featureFlags.enableMaxRecordsPerFile` (since 5.4.0)	Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.
`validations.*`	Optional. Criteria to validate events against
`validations.minimumTimestamp`	This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.
`featureFlags.*`	Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.
`featureFlags.legacyMessageFormat`	This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.
`featureFlags.truncateAtomicFields` (since 5.4.0)	Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.

License​

License