Skip to main content

Manage data structures with Git

Data governance
  • Introduction

  • Create a local data structure

  • Modify, validate and publish

  • Automate with Github Actions

  • Verify GitHub setup

  • Follow up with Data Products

  • Summary

Last updated on

Create a local data structure

Firstly we'll need a place to put things.

$ mkdir -p snowplow-structures/data-structures
$ cd snowplow-structures
tip

snowplow-cli data structures commands default to looking for data structures in ./data-structures.

Now let's create our data structure. We'll create a custom event called 'login'.

$ snowplow-cli ds generate login --vendor com.example
note

ds is an alias for data-structures.

This should provide us the following output

3:00PM INFO generate wrote=data-structures/com.example/login.yaml

The generated file is written to our default data-structures directory under a sub directory matching the --vendor we supplied with a filename that mirrors the name we gave the data structure. Help for all the arguments available to generate is available by running snowplow-cli ds generate --help.

note

This directory layout and file naming scheme is also followed by the download command.

Let's see what it has created for us.

data-structures/com.example/login.yaml
apiVersion: v1
resourceType: data-structure
meta:
hidden: false
schemaType: event
customData: {}
data:
$schema: http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#
self:
vendor: com.example
name: login
format: jsonschema
version: 1-0-0
type: object
properties: {}
additionalProperties: false
  • apiVersion should always be v1
  • resourceType should remain data-structure
  • meta.hidden directly relates to showing and hiding in BDP Console UI
  • meta.schemaType can be event or entity
  • meta.customData is a map of strings to strings that can be used to send across any key/value pairs you'd like to associate with the data structure
  • data is the actual snowplow self describing schema that this data structure describes