Skip to main content

jqFilter

note

This transformation was added in version 3.0.0.

jq is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the gojq package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the tutorial, and full reference manual. This open-source jq playground tool may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are some small differences between jq and gojq.

jqFilter filters messages based on the output of a jq command which is run against the data. The provided command must return a boolean result. false filters the message out, true keeps it.

If the provided jq command returns a non-boolean value error, or results in an error, then the message will be considered invalid, and will be sent to the failure target.

Configuration options​

This example filters out all data that doesn't have an app_id key.

Minimal configuration:

assets/docs/configuration/transformations/builtin/jqFilter-minimal-example.hcl
loading...

Every configuration option:

assets/docs/configuration/transformations/builtin/jqFilter-full-example.hcl
loading...

Helper Functions​

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

  • epoch converts a Go time.Time timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as time.Time. This function can be chained with jq native functions to get past this limitation. For example:
{ foo: .collector_tstamp | epoch | todateiso8601 }
  • epochMillis converts a Go time.Time timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the epoch function truncates to seconds, and the epochMillis function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.
{ foo: .collector_tstamp | epochMillis }