Skip to main content

Kinesis Source

note

To use this source, you need the AWS-specific version of Snowbridge that can only be run on AWS. See the page on Snowbridge distributions for more information.

Authentication​

Authentication is done via the AWS authentication environment variables. Optionally, you can use the role_arn option to specify an ARN to use on the stream.

Setup​

The AWS kinesis source requires the additional setup of a set of dynamoDB tables for checkpointing purposes. To set up a kinesis source, you will need to:

  1. Configure the above required variables in the HCL file.
  2. Create three DynamoDB tables which will be used for checkpointing the progress of the replicator on the stream (Note: details below)

Under the hood we are using a fork of the Kinsumer library which has defined this DynamoDB table structure - these tables need to be created by hand before the application can launch.

TableNameDistKey
${SOURCE_KINESIS_APP_NAME}_clientsID (String)
${SOURCE_KINESIS_APP_NAME}_checkpointsShard (String)
${SOURCE_KINESIS_APP_NAME}_metadataKey (String)

Assuming your AWS credentials have sufficient permission for Kinesis and DynamoDB, your consumer should now be able to run when you launch the executable.

Configuration options​

Here is an example of the minimum required configuration:

assets/docs/configuration/sources/kinesis-minimal-example.hcl
loading...

Here is an example of every configuration option:

assets/docs/configuration/sources/kinesis-full-example.hcl
loading...