# Transformer Kinesis configuration reference
> Configure Transformer Kinesis with stream settings, S3 output, windowing, and monitoring for AWS real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/configuration-reference/
> Documentation index: https://docs.snowplow.io/llms.txt

The configuration reference in this page is written for Transformer Kinesis `6.10.0`

An example of the minimal required config for the Transformer Kinesis can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hocon
license: {
  accept: true
}
```

| Parameter                                                    | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.appName`                                              | Optional. KCL app name. The default is `snowplow-rdb-transformer`                                                                                                                                                                                                                                                                                                                                             |
| `input.streamName`                                           | Required for `kinesis`. Enriched Kinesis stream name.                                                                                                                                                                                                                                                                                                                                                         |
| `input.region`                                               | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `input.position`                                             | Optional. Kinesis position: `LATEST` or `TRIM_HORIZON`. The default is `LATEST`.                                                                                                                                                                                                                                                                                                                              |
| `windowing`                                                  | Optional. Frequency to emit shredding complete message. The default is `10 minutes`. Maximum allowed value is `60 minutes`                                                                                                                                                                                                                                                                                    |
| `output.path`                                                | Required. S3 URI of the transformed output.                                                                                                                                                                                                                                                                                                                                                                   |
| `output.compression`                                         | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                                                                                                                                     |
| `output.region`                                              | AWS region of the S3 bucket. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                          |
| `output.maxRecordsPerFile` (since 5.4.0)                     | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.type` (since 5.4.0)                              | Optional. Either `kinesis` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                                                                                                                                   |
| `output.bad.streamName` (since 5.4.0)                        | Required if output type is `kinesis`. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                                                                                                                 |
| `output.bad.region` (since 5.4.0)                            | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `output.bad.recordLimit` (since 5.4.0)                       | Optional, default = 500. Limits the number of events in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                                  |
| `output.bad.byteLimit` (since 5.4.0)                         | Optional, default = 5242880. Limits the number of bytes in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                               |
| `output.bad.backoffPolicy.minBackoff` (since 5.4.0)          | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                     |
| `output.bad.backoffPolicy.maxBackoff` (since 5.4.0)          | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                           |
| `output.bad.backoffPolicy.maxRetries` (since 5.4.0)          | Optional, default = 10. Maximum number of retries for internal Kinesis errors.                                                                                                                                                                                                                                                                                                                                |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 5.4.0) | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails in case of throughput exceeded.                                                                                                                                                                                                                                                                           |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 5.4.0) | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails in case of throughput exceeded. Writing is retried forever.                                                                                                                                                                                                                                                     |
| `queue.type`                                                 | Required. Type of the message queue. Can be either `sqs` or `sns`.                                                                                                                                                                                                                                                                                                                                            |
| `queue.queueName`                                            | Required if queue type is `sqs`. Name of the SQS queue. SQS queue needs to be FIFO.                                                                                                                                                                                                                                                                                                                           |
| `queue.topicArn`                                             | Required if queue type is `sns`. ARN of the SNS topic.                                                                                                                                                                                                                                                                                                                                                        |
| `queue.region`                                               | AWS region of the SQS queue or SNS topic. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                             |
| `formats.*`                                                  | Schema-specific format settings.                                                                                                                                                                                                                                                                                                                                                                              |
| `formats.transformationType`                                 | Required. Type of transformation, either `shred` or `widerow`. See [Shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) and [Wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format).                                                                             |
| `formats.fileFormat`                                         | Optional. The default is `JSON`. Output file format produced when transformation is `widerow`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                                                                                                    |
| `formats.default`                                            | Optional. The default is `TSV`. Data format produced by default when transformation is `shred`. Either `TSV` or `JSON`. `TSV` is recommended as it enables table autocreation, but requires an Iglu Server to be available with known schemas (including Snowplow schemas). `JSON` does not require an Iglu Server, but requires Redshift JSONPaths to be configured and does not support table autocreation. |
| `formats.tsv`                                                | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `JSON` this list of schemas will still be shredded into `TSV`.                                                                                                                                                                                                                                    |
| `formats.json`                                               | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `TSV` this list of schemas will still be shredded into `JSON`.                                                                                                                                                                                                                                    |
| `formats.skip`                                               | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. Schemas for which loading can be skipped.                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.*`                                       | Send metrics to a StatsD server or stdout.                                                                                                                                                                                                                                                                                                                                                                    |
| `monitoring.metrics.statsd.*`                                | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.hostname`                         | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                                                                                                                                                                                                            |
| `monitoring.metrics.statsd.port`                             | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                                                                                                                                                                                                                     |
| `monitoring.metrics.statsd.tags`                             | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.prefix`                           | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.stdout.*`                                | Optional. For sending metrics to stdout.                                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.stdout.prefix`                           | Optional. Overrides the default metric prefix.                                                                                                                                                                                                                                                                                                                                                                |
| `telemetry.disable`                                          | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                     |
| `telemetry.userProvidedId`                                   | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.dsn`                                      | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                    |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0)         | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                                                                                                                                               |
| `validations.*`                                              | Optional. Criteria to validate events against                                                                                                                                                                                                                                                                                                                                                                 |
| `validations.minimumTimestamp`                               | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                                                                                                                                            |
| `featureFlags.*`                                             | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                                                                                                                                                   |
| `featureFlags.legacyMessageFormat`                           | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                                                                                                                                          |
| `featureFlags.truncateAtomicFields` (since 5.4.0)            | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                                                                                                                                                  |
