This is a complete list of the options that can be configured in the postgres loader's HOCON config file. The example configs in github show how to prepare an input file.
|Required. Can be "Kinesis", "PubSub" or "Local". Configures where input events will be read from.
input.type is Kinesis. Name of the Kinesis stream to read from.
input.type is Kinesis. AWS region in which the Kinesis stream resides.
|Optional. Used when
input.type is Kinesis. Use "TRIM_HORIZON" (the default) to start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. Or use "LATEST" to start streaming just after the most recent record in the shard.
input.type is Kinesis, this sets the polling mode for retrieving records. Can be "FanOut" (the default) or "Polling".
|Optional. Used when
input.retrievalMode.type is "Polling". Configures how many records are fetched in each poll of the kinesis stream. Default 10000.
input.type is PubSub. The name of your GCP project.
input.type is PubSub. Id of the PubSub subscription to read events from
input.type is Local. Path for event source. It can be directory or file. If it is directory, all the files under given directory will be read recursively. Also, given path can be both absolute path or relative path w.r.t. executable.
|Required. Hostname of the postgres database.
|Optional. Port number of the postgres database. Default 5432.
|Required. Name of the postgres database.
|Required. Postgres role name to use when connecting to the database
|Required. Password for the postgres user.
|Required. The Postgres schema in which to create tables and write events.
|Optional. Configures how the client and server agree on ssl protection. Default "REQUIRE"
|Optional. Can be "Kinesis", "PubSub", "Local" or "Noop". Configures where bad rows will be sent. Default is "Noop" which means bad rows will be discarded
bad.type is Kinesis. Name of the Kinesis stream to write to.
bad.type is Kinesis. AWS region in which the Kinesis stream resides.
bad.type is PubSub. The name of your GCP project.
bad.type is PubSub. Id of the PubSub topic to write bad rows to
bad.type is Local. Path of the file to write bad rows
|Optional. Set this to "ENRICHED_EVENTS" (the default) when reading the stream of enriched events in tsv format. Set this to "JSON" when reading a stream of self-describing json, e.g. snowplow bad rows.
|Optional boolean, with default true. For kinesis input, this is used to disable sending metrics to cloudwatch.
We believe these advanced options are set to sensible defaults, and hopefully you won't need to ever change them.
|If producer (PubSub or Kinesis) fails to send item, it will retry to send it again. This field configures backoff time for first retry. Every retry will double the backoff time of previous one.
|Maximum backoff time for retry. After this value is reached, backoff time will no more increase.
input.type is Kinesis. Determines the max number of records to aggregate before checkpointing the records. Default is 1000.
input.type is Kinesis. Determines the max amount of time to wait before checkpointing the records. Default is 10 seconds.
input.type is PubSub. The max number of concurrent evaluation for checkpointer.
|Maximum number of connections database pool is allowed to reach. Default 10
|Size of the thread pool for blocking database operations. Default is value of "maxConnections"
|Set the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. Default 200 milliseconds
|A batch of messages will be emitted when the number of events in batch reaches the given size. Default 500
|A batch of messages will be emitted when the size of the batch reaches the given size. Default 5 MB