Skip to main content

jq

note

This transformation was added in version 3.0.0.

jq is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the gojq package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the tutorial, and full reference manual. This open-source jq playground tool may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are some small differences between jq and gojq.

jq runs a jq command on the message data, and outputs the result of the command. While jq supports multi-element results, commands must output only a single element - this single element can be an array data type.

If the provided jq command results in an error, the message will be considered invalid, and will be sent to the failure target.

The minimal example here returns the input data as a single element array, and the full example maps the data to a new data structure.

The jq transformation will remove any keys with null values from the data.

Configuration options​

Minimal configuration:

assets/docs/configuration/transformations/builtin/jq-minimal-example.hcl
loading...

Every configuration option:

assets/docs/configuration/transformations/builtin/jq-full-example.hcl
loading...

Helper functions​

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

  • epoch converts a Go time.Time timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as time.Time. This function can be chained with jq native functions to get past this limitation. For example:
{ foo: .collector_tstamp | epoch | todateiso8601 }
  • epochMillis converts a Go time.Time timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the epoch function truncates to seconds, and the epochMillis function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.
{ foo: .collector_tstamp | epochMillis }