Snowbridge jqFilter transformation

note

This transformation was added in version 3.0.0.

jq is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the gojq package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the tutorial, and full reference manual. This open-source jq playground tool may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are some small differences between jq and gojq.

jqFilter filters messages based on the output of a jq command which is run against the data. The provided command must return a boolean result. false filters the message out, true keeps it.

If the provided jq command returns a non-boolean value error, or results in an error, then the message will be considered invalid, and will be sent to the failure target.

Configuration options

This example filters out all data that doesn't have an app_id key.

Minimal configuration:

hclassets/docs/configuration/transformations/builtin/jqFilter-minimal-example.hcl

loading...

View on GitHub

Every configuration option:

hclassets/docs/configuration/transformations/builtin/jqFilter-full-example.hcl

loading...

View on GitHub

Filtering examples

The following examples demonstrate common filtering patterns for Snowplow enriched events. You can filter Snowplow enriched events based on any field in the data.

Match an atomic field to any value from a list:

hcl

transform {
  use "jqFilter" {

    # Keep only web and mobile data
    jq_command = <<JQEOT
    .platform | IN("web", "mobile")
JQEOT

    snowplow_mode = true
  }
}

Regex match against a singular entity:

hcl

transform {
  use "jqFilter" {

    # Keep only "environment" matching a regex in custom event
    # `// ""` is needed as `null` is not regex compatible
    jq_command = <<JQEOT
    .contexts_com_acme_env_context_1.environment // "" | test("^prod")
JQEOT

    snowplow_mode = true
  }
}

Filter where any entity entry matches the condition:

hcl

transform {
  use "jqFilter" {

    # Keep if any entry's environment matches one of two values:
    jq_command = <<JQEOT
    .contexts_com_acme_env_context_1 | any(.[]; .environment == "prod" or .environment == "staging")
JQEOT

    snowplow_mode = true
  }
}

Filter for an exact match on a self-describing event field:

hcl

transform {
  use "jqFilter" {
    # Keep only "sku" of "test-data" in custom event
    jq_command = <<JQEOT
    .unstruct_event_com_acme_my_custom_event_1.sku == "test-data"
JQEOT

    snowplow_mode = true
  }
}

Helper Functions

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

• epoch converts a Go time.Time timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as time.Time. This function can be chained with jq native functions to get past this limitation. For example:

{ foo: .collector_tstamp | epoch | todateiso8601 }

• epochMillis converts a Go time.Time timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the epoch function truncates to seconds, and the epochMillis function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.

{ foo: .collector_tstamp | epochMillis }

• hash(algorithm, salt) hashes the input value. To use unsalted hash, pass an empty string for salt. Salt may be provided as an environment variable using hcl syntax.

The following hash algorithms are supported:

• sha1 - SHA-1 hash (160 bits)
• sha256 - SHA-256 hash (256 bits)
• md5 - MD5 hash (128 bits)

{ foo: .user_id |  hash("sha1"; "${env.SHA1_SALT}") }