Skip to main content

Managing data structures via the CLI

info
This documentation only applies to Snowplow BDP. See the feature comparison page for more information about the different Snowplow offerings.

The data-structures subcommand of Snowplow CLI provides a collection of functionality to ease the integration of custom development and publishing workflows.

Snowplow CLI Prerequisites​

Download​

Releases can be found on github https://github.com/snowplow-product/snowplow-cli/releases.

For systems with curl available the following commands should get you started with the latest version. Take care to replace darwin_arm64 with the correct architecture for your system.

curl -L -o snowplow-cli https://github.com/snowplow-product/snowplow-cli/releases/latest/download/snowplow-cli_darwin_arm64
chmod u+x snowplow-cli
info

The following examples assume you remain in the folder containing snowplow-cli.

Configure​

You will need three values.

API Key Id and API Key Secret are generated from the credentials section in BDP Console.

Organization Id can be retrieved from the URL immediately following the .com when visiting BDP console:

Snowplow CLI can take its configuration from a variety of sources. More details are available from ./snowplow-cli data-structures --help. Variations on these three examples should serve most cases.

SNOWPLOW_CONSOLE_API_KEY_ID=********-****-****-****-************
SNOWPLOW_CONSOLE_API_KEY=********-****-****-****-************
SNOWPLOW_CONSOLE_ORG_ID=********-****-****-****-************

Available commands​

Creating data structures​

./snowplow-cli ds generate login_click ./folder-name

Will create a minimal data structure template in a new file ./folder-name/login_click.yaml. Note that you will need to add a vendor name to the template before it will pass validation. Alternatively supply a vendor at creation time with the --vendor com.acme flag.

Downloading data structures​

./snowplow-cli ds download

This command will retrieve all organization data structures. By default it will create a folder named data-structures in the current working directory to put them in. It uses a combination of vendor and name to further break things down.

Given a data structure with vendor: com.acme and name: link_click and assuming the default format of yaml the resulting folder structure will be ./data-structures/com.acme/link_click.yaml.

Validating data structures​

./snowplow-cli ds validate ./folder-name

This command will find all files under ./folder-name (if omitted then ./data-structures) and attempt to validate them using BDP console. It will assert the following

  1. Is each file a valid format (yaml/json) with expected fields
  2. Does the schema in the file conform to snowplow expectations
  3. Given the organization's loading configuration will any schema version number choices have a potentially negative effect on data loading

If any validations fail the command will report the problems to stdout and exit with status code 1.

Publishing data structures​

./snowplow-cli ds publish dev ./folder-name

This command will find all files under ./folder-name (if omitted then ./data-structures) and attempt to publish them to BDP console in the environment provided (dev or prod).

Publishing to dev will also cause data structures to be validated with the validate command before upload. Publishing to prod will not validate but requires all data structures referenced to be present on dev.