# Snowplow Documentation

> Authoritative Snowplow documentation for implementing event tracking, validation, enrichment, governance, and delivery of clean event-level behavioral data. Focus areas include composable analytics, composable CDP, in-product personalization, AI agentic applications, and feeding AI-ready real-time data into warehouses, lakes, streams, and real-time tools.

Documentation for previous versions of components is available on the site but is not included in this file.

---

# Manage your Snowplow account using the Credentials API
> Manage your Snowplow account configuration, users, and API keys through Console, including instructions for obtaining JWT tokens via the Credentials API.
> Source: https://docs.snowplow.io/docs/account-management/

You can control Snowplow Console (e.g., to automate certain actions) through its [API](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml). To use this API, you need to first obtain an API token.

## Create an API key

In [Console](https://console.snowplowanalytics.com/), navigate to **Settings** > **Manage organization** > **API keys for managing Snowplow**. To view this page, you need a _View API keys_ permission.

![API keys generation view](/assets/images/accessing-generated-api-keys-8dd552f45cdaec4af7a5a070c498786e.png)

You can create multiple API keys. It's also possible to delete any key.

When you create an API key, store the _API key ID_ and the _API key_ itself in a secure location. This pair works like a combination of a username and password, and you should treat it with the same level of security.

> **Warning:** All API keys have admin privileges. Do not share these keys with people or systems you do not trust.

## Obtain an access token

Once you have an API key and key ID, you can exchange them for a temporary access token valid for 24 hours.

For example, using curl, the process would look like this:

```bash
curl \

  --header 'X-API-Key-ID: <API_KEY_ID>' \

  --header 'X-API-Key: <API_KEY>' \

  https://console.snowplowanalytics.com/api/msc/v1/organizations/<ORGANIZATION_ID>/credentials/v3/token
```

**Previous versions**

A previous version of the token exchange endpoint is still available, only requiring the API key:

```bash
curl \

  --header 'X-API-Key: <API_KEY>' \

  https://console.snowplowanalytics.com/api/msc/v1/organizations/<ORGANIZATION_ID>/credentials/v2/token
```

This endpoint is deprecated and will be removed in the future. Use the v3 endpoint detailed above instead.

You can find your Organization ID [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

The curl command above will return a JWT as follows:

```json
{ "accessToken": "<JWT>" }
```

## Use the access token with Console API

You can use the access token to supply authorization headers for subsequent API requests:

```bash
curl --header 'Authorization: Bearer <JWT>'
```

---

# Available Console user permissions and roles
> Configure user permissions in Snowplow Console with Global Admin, User, and Custom roles to control access to environments, data structures, tracking plans, data models, and API keys.
> Source: https://docs.snowplow.io/docs/account-management/managing-permissions/

To set a user's permissions, navigate to **Settings** > **Users** and then to the user whose account you'd like to manage.

## What permissions can be set?

Snowplow Console sets permissions for each area of Console as summarized below:

| **Console feature** | **Description**                                                                                                            | **Possible permissions**                                     |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| User management     | The management and addition of user access. This permission cannot be configured on a Custom role.                         | No access Edit Create                                        |
| Environments        | The management of pipeline and development environments. This includes managing which Enrichments run on each environment. | No access View Edit                                          |
| Tracking plans      | The management and creation of tracking plans.                                                                             | No access View Edit Create                                   |
| Data structures     | The management and creation of the schemas that define the events and entities you are capturing.                          | No access View Edit on development Edit on production Create |
| Data models         | The management and creation of your data models.                                                                           | No access View Edit Create                                   |
| API keys            | The management and creation of API keys.                                                                                   | No access View Manage                                        |

## How are permissions set?

To set permissions for a user, navigate to **Settings** > **Users** and select the user, within the management screen for their user you will be able to set their permissions.

There are three ways of setting user permissions:

- Global Admin (pre-defined role)
- User (pre-defined role)
- Custom (custom permissions role)

The following tables describe the default permissions for each role.

#### User permission set

| **Console feature** | **Permissions** |
| ------------------- | --------------- |
| Environments        | View access     |
| User management     | View access     |
| Tracking plans      | View access     |
| Data structures     | View access     |
| Data models         | View access     |
| API keys            | View access     |

#### Global Admin permission set

| **Console feature** | **Permissions** |
| ------------------- | --------------- |
| User management     | Full access     |
| Environments        | Full access     |
| Tracking plans      | Full access     |
| Data structures     | Full access     |
| Data models         | Full access     |
| API keys            | Full access     |

#### Custom permission set

| **Console feature** | **Permissions**             |
| ------------------- | --------------------------- |
| User management     | Customized by you, per user |
| Environments        | Customized by you, per user |
| Tracking plans      | Customized by you, per user |
| Data structures     | Customized by you, per user |
| Data models & jobs  | Customized by you, per user |
| API keys            | Customized by you, per user |

A note on API keys and permissions

Please note:

1. Any API keys you create have full admin permissions

2. Any existing Iglu API keys allow permissions to be side-stepped by connecting directly to Iglu servers

The recommended approach is to remove all existing API keys and Iglu keys, and set the API keys permission respectively so that only trusted users can create new keys.

## What does each permission mean?

### Environments

An environment is the collective name for your Production pipelines, QA pipelines and development environments.

An environment has three permissions:

- **No access** - the user will not see the environment management screens.
- **View** - the user can see the environment management screen, but cannot edit anything. This is the default setting for the User role.
- **Edit** - the user can make edits to the environment. This includes configuration such as enrichment enablement, enrichment configuration and collector configuration.

### Tracking plans

Tracking plans have four permissions:

- **No access** - the user will not see the tracking plan management screens.
- **View** - the user can see the tracking plan management screens, but cannot edit anything. This permission and all tracking plan permissions below require the user to have at least the **View** permission on data structures.
- **Edit** - the user can see the tracking plan management screens, and can make edits to existing tracking plans.
- **Create** - the user can create new tracking plans.

### Data structures

Data structures have five permissions:

- **No access** - the user will not see the data structure management screen.
- **View** - the user can see the data structure management screen, but cannot edit anything.
- **Edit on development** - the user can see the data structure management screen, and can make edits to data structures but only publish them to the development registry.
- **Edit on production** - the user can see the data structure management screen, and can make edits to data structures, and can publish changes to the production registry.
- **Create** - the user can create new data structures.

### Data models

Data models and jobs have four permissions:

- **No access** - the user will not see the data model management screens.
- **View** - the user can see the data model management screens, but cannot edit anything. This is the default setting for the User role.
- **Edit** - the user can see the data model management and can make edits to data models in production. This is the default setting for the Global Admin role.
- **Create** - the user can create new data models.

### API keys

API keys have four permissions:

- **No access** - the user will not see the API key management screens.
- **View** - the user can see the API key descriptions but cannot see the keys themselves or manage them.
- **Manage** - the user can see and manage the API keys.
- **Create** - the user can generate new API keys.

## Troubleshooting

You shouldn’t be required to logout for new permissions to take effect, but if you do find permissions aren’t applying as requested logging out and back in should force the new permissions to apply.

---

# Manage users in Console directly or with SSO
> Add and remove users in Snowplow Console, configure Single Sign-On with identity providers including Google Workspace, Entra ID, Okta, and OpenID Connect.
> Source: https://docs.snowplow.io/docs/account-management/managing-users/

There are two ways to add and remove users in Console: directly managed in Console, or managed through your Single Sign-On (SSO) provider.

SSO is an authentication process that allows users to access multiple applications after signing in to a central Identity Provider. Snowplow supports SSO integration for the majority of identity providers.

For organizations **not using SSO**, users can be added and removed directly in Console by navigating to **Settings** > **Users** in the navigation and creating a new user, or removing an existing user from there. Newly added users will receive an email to set their password and will be added with limited permissions, which you can then widen.

For organizations **using SSO**, you will need to configure your account with your Identity Provider before you can add or remove users.

## SSO permissions

Only system administrators can set up SSO for their company. For information on setting permissions for individual users, see [Managing user permissions](/docs/account-management/managing-permissions/).

## How to enable SSO for your account

Setting up SSO for your account requires some information to be exchanged between you as the Identity Provider and Snowplow as the Service Provider. Depending on your Identity Provider, the information that is required is slightly different.

To enable single sign-on (SSO) for Snowplow, follow these steps inside Console:

1. Go to the [manage organization](https://console.snowplowanalytics.com/settings) page.
2. Select [Single sign-on (SSO)](https://console.snowplowanalytics.com/users) from the Users panel. The SSO configuration is only visible to users with Admin role.
3. Click **Continue** and follow the steps for your Identity Provider.

## Which Identity Providers (IdPs) are supported?

Snowplow’s SSO capability enables connections with many IdPs, including: 

- ADFS
- Auth0
- Entra ID (formerly known as Azure AD)
- Google Workspace
- Keycloak
- Okta
- PingFederate

Because Snowplow supports OpenID Connect and SAML, virtually any external Identity Provider that uses those standards should work.

## What information will you need from us?

This will differ depending on your Identity Provider, but typically will include information such as:

- **Entity ID** - the URL that identifies the identity provider issuing a SAML request, this will be specific to your identity provider.
- **Metadata URL** - the URL that allows access to obtain SSO configuration data, this will be specific to your identity provider.
- **Redirect Login URL** - the URL where users in the company sign in to the identity provider.
- **User information mapping** - locations of information required by Snowplow Console such as first name, last name and, optionally, job title.

## What happens when SSO is enabled?

### Adding new users

Snowplow supports just-in-time provisioning with SSO connections. When a user logs in for the first time, a corresponding user account with the same email is created in Snowplow.

A new user created via SSO will have a custom permissions set that allows them to view-only, as outlined below. This can then be edited by anyone with the Global Admin role on your account. For more details on setting user access, see [Managing user permissions](/docs/account-management/managing-permissions/).

### Existing users

If a user already has a Snowplow account prior to SSO being enabled, the two accounts will be merged, and the user's current permissions will be applied.

### Logging in 

When SSO is enabled, anybody who signs into Snowplow Console with an email address that uses your specified domain will be authenticated via SSO and your Identity Provider.

Once SSO is enabled, users on your domain can no longer sign in with their old email address and password, or manage their personal details or password as these will all be managed within your Identity Provider.

## Disabling SSO

If your company enables SSO, and later decides to disable it:

- Users who did not set up a password before SSO was enabled must click Reset password on the login page to obtain a password.
- Users who set up a password before SSO was enabled can log in with their old username and password.

---

# Go Analytics SDK
> Go library for parsing Snowplow enriched events with efficient field access and transformation to JSON or maps for serverless functions.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-go/

## 1. Overview

The [Snowplow Analytics SDK for Go](https://github.com/snowplow/snowplow-golang-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your Go event processing, data modeling and machine-learning jobs. You can use this SDK with [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions/), [Google App Engine](https://cloud.google.com/appengine) and other Golang-compatible data processing frameworks.

## 2. Compatibility

Snowplow Analytics SDK fo Go was tested with Go versions 1.13-1.15.

There are only two external dependencies currently:

- `github.com/json-iterator/go` - used to parse JSON

- `github.com/pkg/errors` - used to provide an improvement on the standard error reporting.

## 3. Setup

snowplow/snowplow-golang-analytics-sdk can be imported to a project as a go module:

`go get github.com/snowplow/snowplow-golang-analytics-sdk`

## 4. Usage

### 4.1 Overview

The [Snowplow Analytics SDK for Go](https://github.com/snowplow/snowplow-golang-analytics-sdk) provides you an API to parse an enriched event from it's TSV-string form to a `ParsedEvent` slice of strings, then a set of methods to transform the entire event or a subset of fields into either `JSON` or `map` form. It also offers methods to efficiently get a field from the `ParsedEvent`.

### 4.2 Summary of example usage

```bash
go get github.com/snowplow/snowplow-golang-analytics-sdk
```

```go
import "github.com/snowplow/snowplow-golang-analytics-sdk/analytics"





parsed, err := ParseEvent(event) // Where event is a valid TSV string Snowplow event.

if err != nil {

  fmt.Println(err)

}



parsed.ToJson() // whole event to JSON

parsed.ToMap() // whole event to map

parsed.GetValue("page_url") // get a value for a single canonical field

parsed.GetSubsetMap("page_url", "domain_userid", "contexts", "derived_contexts") // Get a map of values for a set of canonical fields

parsed.GetSubsetJson("page_url", "unstruct_event") // Get a JSON of values for a set of canonical fields
```

### 4.3 API

```go
func ParseEvent(event string) (ParsedEvent, error)
```

ParseEvent takes a Snowplow Enriched event tsv string as input, and returns a 'ParsedEvent' typed slice of strings. Methods may then be called on the resulting ParsedEvent type to transform the event, or a subset of the event to Map or Json.

```go
func (event ParsedEvent) ToJson() ([]byte, error)
```

ToJson transforms a valid Snowplow ParsedEvent to a JSON object.

```go
func (event ParsedEvent) ToMap() (map[string]interface{}, error)
```

ToMap transforms a valid Snowplow ParsedEvent to a Go map.

```go
func (event ParsedEvent) GetSubsetJson(fields ...string) ([]byte, error)
```

GetSubsetJson returns a JSON object containing a subset of the event, containing only the atomic fields provided, without processing the rest of the event.

For custom events and contexts, only "unstruct\_event", "contexts", or "derived\_contexts" may be provided, which will produce the entire data object for that field.

For contexts, the resultant map will contain all occurrences of all contexts within the provided field.

```go
func (event ParsedEvent) GetSubsetMap(fields ...string) (map[string]interface{}, error)
```

GetSubsetMap returns a map of a subset of the event, containing only the atomic fields provided, without processing the rest of the event.

For custom events and entites, only "unstruct\_event", "contexts", or "derived\_contexts" may be provided, which will produce the entire data object for that field.

For contexts, the resultant map will contain all occurrences of all contexts within the provided field.

```go
func (event ParsedEvent) GetValue(field string) (interface{}, error)
```

GetValue returns the value for a provided atomic field, without processing the rest of the event. For unstruct\_event, it returns a map of only the data for the unstruct event. For contexts and derived\_contexts, it returns the data for all contexts or derived\_contexts in the event.

```go
func (event ParsedEvent) ToJsonWithGeo() ([]byte, error)
```

ToJsonWithGeo adds the geo\_location field, and transforms a valid Snowplow ParsedEvent to a JSON object.

```go
func (event ParsedEvent) ToMapWithGeo() (map[string]interface{}, error)
```

ToMapWithGeo adds the geo\_location field, and transforms a valid Snowplow ParsedEvent to a Go map.

---

# JavaScript and TypeScript Analytics SDK
> Lightweight JavaScript and TypeScript library to transform Snowplow enriched TSV events into JSON for serverless functions and Node.js.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-javascript/

## Overview

The [Snowplow JavaScript and TypeScript Analytics SDK](https://github.com/snowplow-incubator/snowplow-js-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your JavaScript event processing, data modeling and machine-learning jobs. You can use this SDK with [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions/), [Google App Engine](https://cloud.google.com/appengine) and other JavaScript-compatible frameworks.

## Setup

Install using your preferred package manager, such as npm:

```bash
npm install --save snowplow-analytics-sdk
```

## Usage

### Overview

The [Snowplow JavaScript and TypeScript Analytics SDK](https://github.com/snowplow-incubator/snowplow-js-analytics-sdk) provides you an API to parse an enriched event from it's TSV-string form to a `JSON` string.

### Example

To consume in an AWS lambda you would do something like this in your `app.js`:

```javascript
const { transform } = require('snowplow-analytics-sdk');



module.exports.handler = (input) => {

  let event = transform(

    new Buffer(input.Records[0].kinesis.data, 'base64').toString('utf8'),

  );



  // ...

};
```

Or in `app.ts`:

```javascript
import { transform } from 'snowplow-analytics-sdk';



export function handler(input: any) {

  let event = transform(

    new Buffer(input.Records[0].kinesis.data, 'base64').toString('utf8'),

  );



  // ...

}
```

## API

### `transform(event: string): Event`

- `event: string` - TSV string containing event data.

Returns decoded [Snowplow enriched event](/docs/fundamentals/canonical-event/).

---

# .NET Analytics SDK
> .NET SDK with JSON event transformer for processing Snowplow enriched events in Azure Data Lake Analytics, Azure Functions, and C# applications.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-net/

## 1. Overview

The [Snowplow Analytics SDK for .NET](https://github.com/snowplow/snowplow-dotnet-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your .NET event processing, data modeling and machine-learning jobs. You can use this SDK with [Azure Data Lake Analytics](https://azure.microsoft.com/en-gb/services/data-lake-analytics/), [Azure Functions](https://azure.microsoft.com/en-gb/services/functions/), [AWS Lambda](https://aws.amazon.com/lambda/), [Microsoft Orleans](https://dotnet.github.io/orleans/) and other .NET-compatible data processing frameworks.

The .NET Analytics SDK makes it significantly easier to build applications that consume Snowplow enriched data directly from Event Hubs or Azure Blob Storage.

## 2. Compatibility

Snowplow .NET Analytics SDK targets [.NET Standard 1.3](https://github.com/dotnet/standard/blob/master/docs/versions.md).

## 3. Setup

To add the .NET Analytics as a dependency to your project, install it in the Visual Studio Package Manager Console using [NuGet](https://www.nuget.org/):

```powershell
Install-Package Snowplow.Analytics
```

## 4. Event Transformer

### 4.1 Overview

The Snowplow enriched event is a relatively complex TSV string containing self-describing JSONs. Rather than work with this structure directly, Snowplow analytics SDKs ship with _event transformers_, which translate the Snowplow enriched event format into something more convenient for engineers and analysts.

As the Snowplow enriched event format evolves towards a cleaner [Apache Avro](https://avro.apache.org/)-based structure, we will be updating this Analytics SDK to maintain compatibility across different enriched event versions.

Working with the Snowplow .NET Analytics SDK therefore has two major advantages over working with Snowplow enriched events directly:

1. The SDK reduces your development time by providing analyst- and developer-friendly transformations of the Snowplow enriched event format
2. The SDK futureproofs your code against new releases of Snowplow which update our enriched event format

Currently the Analytics SDK for .NET ships with one event transformer: the JSON Event Transformer.

### 4.2 The JSON Event Transformer

The JSON Event Transformer takes a Snowplow enriched event and converts it into a JSON ready for further processing. This transformer was adapted from the code used to load Snowplow events into Elasticsearch in the Kinesis real-time pipeline.

The JSON Event Transformer converts a Snowplow enriched event into a single JSON like so:

```json
{

  "app_id":"demo",

  "platform":"web",

  "etl_tstamp":"2015-12-01T08:32:35.048Z",

  "collector_tstamp":"2015-12-01T04:00:54.000Z",

  "dvce_tstamp":"2015-12-01T03:57:08.986Z",

  "event":"page_view",

  "event_id":"f4b8dd3c-85ef-4c42-9207-11ef61b2a46e",

  "txn_id":null,

  "name_tracker":"co",

  "v_tracker":"js-2.5.0",

  "v_collector":"clj-1.0.0-tom-0.2.0",...
```

The most complex piece of processing is the handling of the self-describing JSONs found in the enriched event's `unstruct_event`, `contexts` and `derived_contexts` fields. All self-describing JSONs found in the event are flattened into top-level plain (i.e. not self-describing) objects within the enriched event JSON.

For example, if an enriched event contained a `com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`, then the final JSON would contain:

```json
{

  "app_id":"demo",

  "platform":"web",

  "etl_tstamp":"2015-12-01T08:32:35.048Z",

  "unstruct_event_com_snowplowanalytics_snowplow_link_click_1": {

    "targetUrl":"http://www.example.com",

    "elementClasses":["foreground"],

    "elementId":"exampleLink"

  },...
```

### 4.3 Examples

You can convert an enriched event TSV string to a JSON like this:

```csharp
using Snowplow.Analytics.Json;

using Snowplow.Analytics.Exceptions;



try

{

    EventTransformer.Transform(enrichedEventTsv);

}

catch (SnowplowEventTransformationException sete)

{

    sete.ErrorMessages.ForEach((message) => Console.WriteLine(message));

}
```

If there are any problems in the input TSV (such as unparseable JSON fields or numeric fields), the `transform` method will throw a `SnowplowEventTransformationException`. This exception contains a list of error messages - one for every problematic field in the input.

---

# Parse enriched events in Azure Data Lake
> Custom U-SQL extractor for parsing Snowplow enriched events in Azure Data Lake Analytics with direct access to nested context fields.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-net/snowplow-event-extractor/

[Azure Data Lake](https://azure.microsoft.com/en-in/solutions/data-lake/) is a secure and scalable data storage and analytics service. [Azure Data Lake Analytics](https://azure.microsoft.com/en-in/services/data-lake-analytics/) includes [U-SQL](https://blogs.msdn.microsoft.com/visualstudio/2015/09/28/introducing-u-sql-a-language-that-makes-big-data-processing-easy/), a big-data query language for writing queries that analyze data.

## Event Extractor

Snowplow Event Extractor is an ADLA custom extractor that allows you to parse **[Snowplow enriched events](/docs/fundamentals/canonical-event/)**. Snowplow’s enrichment process outputs enriched events in a TSV format consisting of 131 fields.

EventExtractor implements IExtractor interface:

```csharp
[SqlUserDefinedExtractor]

public class EventExtractor : IExtractor

{

    private static readonly string ROW_DELIMITER = '\t';



    public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)

    {

       //split the input based on ROW_DELIMITER

       //set the output data on the output object

       //EventExtractor only outputs columns and values that are defined with the output.

    }

}
```

## Usage

Following is base U-SQL script that uses a Event Extractor:

```sql
DECLARE @input_file string = @"\snowplow\event.tsv";



@rs0 =

    EXTRACT

        app_id string,

        platform string

    FROM @input_file

    USING new Snowplow.EventExtractor();
```

The most complex piece of processing is the handling of the self-describing JSONs found in the enriched event's unstruct\_event, contexts and derived\_contexts fields. Consider contexts found in the tsv:

```json
{

    'schema': 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0',

    'data': [{

        'schema': 'iglu:org.schema/WebPage/jsonschema/1-0-0',

        'data': {

            'genre': 'blog',

            'inLanguage': 'en-US',

            'datePublished': '2014-11-06T00:00:00Z',

            'author': 'Devesh Shetty',

            'breadcrumb': ['blog', 'releases']

        }

    }, {

        'schema': 'iglu:org.w3/PerformanceTiming/jsonschema/1-0-0',

        'data': {

            'navigationStart': 1415358089861,

            'unloadEventStart': 1415358090270,

            'unloadEventEnd': 1415358090287,

            'redirectStart': 0,

            'redirectEnd': 0

        }

    }]

}
```

One of the ways to fetch data from context would be to use user-defined function(UDF):

```sql
DECLARE @input_file string = @"\snowplow\event.tsv";



//extract context from tsv

@rs0 =

    EXTRACT

        context string

    FROM @input_file

    USING new Snowplow.EventExtractor();



/*

context has nested data array

*/

@parseData =

    SELECT Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(context, "data[*]").Values AS data_arr,

    FROM @rs0;



/*

The nested data array inside context consists of an array from which we parse the inner data field

*/

@parseGenre =

    SELECT Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(data_arr, "$.data.genre").Values AS genre,

    FROM @parseData;
```

The above process can get quite complex. So to abstract away the complexity, Snowplow Event Extractor follows a simple mapping:

```sql
DECLARE @input_file string = @"\snowplow\event.tsv";



//extract genre from context directly

@rsGenre =

    EXTRACT

        context.data.genre

    FROM @input_file

    USING new Snowplow.EventExtractor();
```

---

# Python Analytics SDK
> Python SDK for processing Snowplow enriched events with run manifest support for idempotent data processing in PySpark and AWS Lambda.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-python/

## 1. Overview

The [Snowplow Analytics SDK for Python](https://github.com/snowplow/snowplow-python-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your Python event processing, data modeling and machine-learning jobs. You can use this SDK with [Apache Spark](http://spark.apache.org/), [AWS Lambda](https://aws.amazon.com/lambda/), and other Python-compatible data processing frameworks.

## 2. Compatibility

Snowplow Python Analytics SDK was tested with Python of versions: 2.7, 3.3, 3.4, 3.5.

As analytics SDKs supposed to be used heavily in conjunction with data-processing engines such as [Apache Spark](http://spark.apache.org/), our goal is to maintain compatibility with all versions that PySpark supports. Whenever possible we try to maintain compatibility with broader range of Python versions and computing environments. This is achieved mostly by minimazing and isolating third-party dependencies and libraries.

There are only one external dependency currently:

- [Boto3](https://aws.amazon.com/sdk-for-python/) - AWS Python SDK that used to provide access to Event Load Manifests.

These dependencies can be installed from the package manager of the host system or through PyPi.

## 3. Setup

### 3.1 PyPI

The Snowplow Python Analytics SDK is published to [PyPI](https://pypi.python.org/), the official third-party software repository for the Python programming language.

This makes it easy to either install the SDK locally, or to add it as a dependency into your own Python app or Spark job.

### 3.2 pip

To install the Snowplow Python Analytics SDK locally, assuming you already have Pip installed:

```bash
$ pip install snowplow_analytics_sdk --upgrade
```

To add the Snowplow Analytics SDK as a dependency to your own Python app, edit your `requirements.txt` and add:

```text
snowplow_analytics_sdk==0.2.3
```

### 3.3 easy\_install

If you are still using easy\_install:

```bash
$ easy_install -U snowplow_analytics_sdk
```

## 4. Run Manifests

### 4.1 Overview

The [Snowplow Analytics SDK for Python](https://github.com/snowplow/snowplow-python-analytics-sdk) provides you an API to work with run manifests. Run manifests is simple way to mark chunk (particular run) of enriched data as being processed, by for example Apache Spark data-modeling job.

### 4.2 Usage

Run manifests functionality resides in new `snowplow_analytics_sdk.run_manifests` module.

Main class is `RunManifests`, that provides access to DynamoDB table via `contains` and `add`, as well as `create` method to initialize table with appropriate settings. Other commonly-used function is `list_runids` that is gives S3 client and path to folder such as `enriched.archive` or `shredded.archive` from `config.yml` lists all folders that match Snowplow run id format (`run-YYYY-mm-DD-hh-MM-SS`). Using `list_runids` and `RunManifests` you can list job runs and safely process them one by one without risk of reprocessing.

### 4.3 Example

Here's a short usage example:

```python
from boto3 import client

from snowplow_analytics_sdk.run_manifests import *



s3 = client('s3')

dynamodb = client('dynamodb')



dynamodb_run_manifests_table = 'snowplow-run-manifests'

enriched_events_archive = 's3://acme-snowplow-data/storage/enriched-archive/'

run_manifests = RunManifests(dynamodb, dynamodb_run_manifests_table)



run_manifests.create()   # This should be called only once



for run_id in list_runids(s3, enriched_events_archive):

    if not run_manifests.contains(run_id):

        process(run_id)

        run_manifests.add(run_id)

    else:

        pass
```

In above example, we create two AWS service clients for S3 (to list job runs) and for DynamoDB (to access manifests). These clients are provided via [boto3](https://aws.amazon.com/sdk-for-python/) Python AWS SDK and can be initialized with static credentials or with system-provided credentials.

Then we list all run ids in particular S3 path and process (by user-provided `process` function) only those that were not processed already. Note that `run_id` is simple string with S3 key of particular job run.

`RunManifests` class is a simple API wrapper to DynamoDB, using which you can:

- `create` DynamoDB table for manifests,
- `add` run to table
- check if table `contains` run id

---

# Scala Analytics SDK
> Parse Snowplow enriched events into case classes with JSON transformation and event inventory metadata for Apache Spark, Flink, and AWS Lambda.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-scala/

## 1. Overview

The [Snowplow Analytics SDK for Scala](https://github.com/snowplow/snowplow-scala-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your Scala event processing, data modeling and machine-learning jobs. You can use this SDK with [Apache Spark](http://spark.apache.org/), [AWS Lambda](https://aws.amazon.com/lambda/), [Apache Flink](https://flink.apache.org/), [Scalding](https://github.com/twitter/scalding), [Apache Samza](http://samza.apache.org/) and other JVM-compatible data processing frameworks.

The Scala Analytics SDK makes it significantly easier to build applications that consume Snowplow enriched data directly from Kinesis or S3.

## 2. Compatibility

Snowplow Scala Analytics SDK was compiled against Scala versions 2.12 and 2.13. Minimum required Java Runtime is JRE8.

## 3. Setup

The latest version of Snowplow Scala Analytics SDK is 3.0.0 and it is available on Maven Central.

### 3.1 SBT

If you’re using SBT, add the following lines to your build file:

```scala
// Dependency

libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-analytics-sdk" % "3.0.0"
```

Note the double percent (`%%`) between the group and artifactId. This will ensure that you get the right package for your Scala version.

### 3.2 Gradle

If you are using Gradle in your own job, then add following lines in your `build.gradle` file:

```gradle
dependencies {

  ...

  // Snowplow Scala Analytics SDK

  compile 'com.snowplowanalytics:snowplow-scala-analytics-sdk_2.12:3.0.0'

}
```

Note that you need to change `_2.12` to `_2.13` in artifactId if you're using Scala 2.13.

### 3.3 Maven

If you are using Maven in your own job, then add following lines in your `pom.xml` file:

```xml
<dependency>

  <groupId>com.snowplowanalytics</groupId>

  <artifactId>snowplow-scala-analytics-sdk_2.12</artifactId>

  <version>3.0.0</version>

</dependency>
```

Note that you need to change `_2.12` to `_2.13` in artifactId if you're using Scala 2.13.

## 4. Scala Analytics SDK Event Transformer

### 4.1 Overview

The Snowplow enriched event is a relatively complex TSV string containing scalars and self-describing JSONs. Rather than work with this structure directly, Snowplow analytics SDKs ship with _event transformers_, which translate the Snowplow enriched event format into other formats that are more convenient for engineers and analysts.

As the Snowplow enriched event format evolves towards a cleaner [Apache Avro](https://avro.apache.org/)-based structure, we will be updating this SDK to maintain compatibility across different enriched event versions.

Working with the Snowplow Scala Analytics SDK therefore has two major advantages over working with Snowplow enriched events directly:

1. The SDK reduces your development time by providing analyst- and developer-friendly transformations of the Snowplow enriched event format;
2. The SDK futureproofs your code against new releases of Snowplow which update our enriched event format.

Currently the Analytics SDK for Scala ships with one event transformer: the JSON Event Transformer.

### 4.2 The JSON Event Transformer

The JSON Event Transformer takes a Snowplow enriched event and converts it into a JSON ready for further processing. This transformer was adapted from the code used to load Snowplow events into Elasticsearch in the Kinesis real-time pipeline.

The JSON Event Transformer converts a Snowplow enriched event into an instance of the `Event` case class, a representation of a canonical Snowplow event, like so:

```scala
Event(

  app_id = Some("angry-birds"),

  platform = Some("web"),

  etl_tstamp = Some(Instant.parse("2017-01-26T00:01:25.292Z")),

  collector_tstamp = Instant.parse("2013-11-26T00:02:05Z"),

  dvce_created_tstamp = Some(Instant.parse("2013-11-26T00:03:57.885Z")),

  event = Some("page_view"),

  event_id = UUID.fromString("c6ef3124-b53a-4b13-a233-0088f79dcbcb"),

  txn_id = Some(41828),

  name_tracker = Some("cloudfront-1"),

  v_tracker = Some("js-2.1.0"),

  v_collector = "clj-tomcat-0.1.0",

  v_etl = "serde-0.5.2"

  /* ... */

)
```

This case class can be rendered into a JSON object, and subsequently a JSON string, or worked with to interact with the event's fields in a typesafe manner.

The most complex piece of processing is the handling of the self-describing JSONs found in the enriched event's `unstruct_event`, `contexts` and `derived_contexts` fields. Currently there are two alternative behaviors for handling them in the event transformer:

1. Under the original "lossy" behavior, if an enriched event contained a `com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`, then the unstructured event field would be rendered in the final JSON like this:

```json
"unstruct_event_com_snowplowanalytics_snowplow_link_click_1": {

  "targetUrl": "http://www.example.com",

  "elementClasses": ["foreground"],

  "elementId": "exampleLink"

}
```

2. Under the new "lossless" behavior, available since 0.3.1, if an enriched event contained a `com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`, then the final JSON (if turned into a string) would contain a self-describing JSON object instead:

```json
"unstruct_event": {

  "schema": "iglu:com.snowplowanalytics.snowplow/link_click/1-0-1",

  "data": {

    "targetUrl": "http://www.example.com",

    "elementClasses": ["foreground"],

    "elementId": "exampleLink"

  }

}
```

Along with the `Event` case class, the JSON Event Transformer comes with the following functions:

- `Event.parse(line)` - similar to the old `transform` function, this method accepts an enriched Snowplow event as a string and returns an `Event` instance as a result.
- `event.toJson(lossy)` - similar to the old `getValidatedJsonEvent` function, it transforms an `Event` into a validated JSON whose keys are the field names corresponding to the EnrichedEvent POJO of the Scala Common Enrich project. If the lossy argument is true, any self-describing events in the fields (unstruct\_event, contexts and derived\_contexts) are returned in a "shredded" format, e.g. `"unstruct_event_com_acme_1_myField": "value"`. If it is set to false, they are not flattened into underscore-separated top-level fields, using a standard self-describing format instead.
- `event.inventory` - extracts metadata from the event containing information about the types and Iglu URIs of its shred properties (unstruct\_event, contexts and derived\_contexts). Unlike version 0.3.0, it no longer requires a `transformWithInventory` call and can be obtained from any `Event` instance.
- `atomic` - returns the event as a map of keys to Circe JSON values, while dropping inventory fields. This method can be used to modify an event's JSON AST before converting it into a final result.
- `ordered` - returns the event as a list of key/Circe JSON value pairs. Unlike `atomic`, which has randomized key ordering, this method returns the keys in the order of the canonical event model, and is particularly useful for working with relational databases.

#### Event inventory

An event's inventory is simply a list of metadata about its shredded types:

1. Where it was extracted from: `unstruct_event` column (`UnstructEvent`), `contexts` column (`Contexts(CustomContexts)`) or `derived_contexts` column (`Contexts(DerivedContexts)`)
2. Its Iglu URI (e.g. `iglu:com.acme/context/jsonschema/1-0-0`), stored as an Iglu `SchemaKey` instance.

### 4.3 Examples

#### 4.3.1 Using from Apache Spark

The Scala Analytics SDK is a great fit for performing Snowplow [event data modeling](http://snowplowanalytics.com/blog/2016/03/16/introduction-to-event-data-modeling/) in Apache Spark and Spark Streaming.

Here’s the code we use internally for our own data modeling jobs:

```scala
import cats.data.Validated

import com.snowplowanalytics.snowplow.analytics.scalasdk.Event



val events = input.flatMap(line =>

  Event.parse(line) match {

    case Validated.Valid(event) => Some(event.toJson(true).noSpaces)

    case Validated.Invalid(_) => None

  }

)



val dataframe = spark.read.json(events: _*)
```

#### 4.3.2 Using from AWS Lambda

The Scala Analytics SDK is a great fit for performing analytics-on-write, monitoring or alerting on Snowplow event streams using AWS Lambda.

Here’s some sample code for transforming enriched events into JSON inside a Scala Lambda:

```scala
import com.snowplowanalytics.snowplow.analytics.scalasdk.Event



def recordHandler(event: KinesisEvent) {



  val events = for {

    rec <- event.getRecords

    line = new String(rec.getKinesis.getData.array())

    event = Event.parse(line)

  } yield event



  /* ... */

}
```

---

# Analytics SDKs for event data transformation
> Transform Snowplow enriched TSV events into JSON for data modeling and machine learning in Scala, JavaScript, Go, Python, and .NET.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/

The Snowplow Analytics SDKs are designed for data engineers and data scientists working with Snowplow in a number of languages.

Some good use cases for the SDKs include:

1. Transforming the Enriched TSV to Enriched JSON for further processing
2. Developing AI/ML models on your event data
3. Performing analytics-on-write in AWS Lambda as part of our Kinesis real-time pipeline
4. Within Snowplow pipeline components to process event data

## Snowplow Analytics SDKs

- [Scala Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-scala/) - lets you work with Snowplow enriched events in your Scala event processing, data modeling and machine-learning jobs. You can use this SDK with Apache Spark, AWS Lambda, GCP Cloud Functions, Apache Flink and other Scala-compatible data processing frameworks.
- [JavaScript and TypeScript Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-javascript/) - lets you work with Snowplow enriched events in your Node.js or other JavaScript environments. This SDK can be used with AWS Lambda and Google Cloud Functions.
- [Go Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-go/) - lets you work with Snowplow enriched events in your Go environments. This SDK can be used with AWS Lambda and Google Cloud Functions.
- [Python Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-python/) - lets you work with Snowplow enriched events in your Python event processing, data modeling and machine-learning jobs. You can use this SDK with Apache Spark, AWS Lambda, GCP Cloud Functions and other Python-compatible data processing frameworks.
- [.NET Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-net/) - lets you work with Snowplow enriched events in your .NET event processing, data modeling and machine-learning jobs. You can use this SDK with Azure Data Lake Analytics, Azure Function, AWS Lambda, GCP Cloud Functions other .NET-compatible data processing frameworks.

---

# Dataflow Runner for AWS EMR clusters
> CLI tool for launching and managing AWS EMR clusters with templated playbooks for Hadoop and Spark jobs with distributed locking support.
> Source: https://docs.snowplow.io/docs/api-reference/dataflow-runner/

Dataflow Runner is a system for creating and running [AWS EMR](https://aws.amazon.com/emr/) jobflow clusters and steps. It uses templated playbooks to define your cluster, and the Hadoop/Spark/et al jobs that you want to run.

### Installation

- Platform native binaries are available from the GitHub releases [page](https://github.com/snowplow/dataflow-runner/releases).
- Docker images are available at [DockerHub](https://hub.docker.com/r/snowplow/dataflow-runner) as of version `0.7.3`.

### Cluster Configuration

A cluster configuration contains all of the information needed to create a new cluster which is ready to accept a playbook. Currently AWS EMR is the only supported data-flow fabric.

For the cluster template see: [config/cluster.json.sample](https://github.com/snowplow/dataflow-runner/blob/master/config/cluster.json.sample)

### Playbook Configuration

A playbook consists of one of more _steps_. Steps are added to the cluster and run in series.

For the playbook template see: [config/playbook.json.sample](https://github.com/snowplow/dataflow-runner/blob/master/config/playbook.json.sample)

### Templates

Configuration files are run through Golang’s [text template processor](http://golang.org/pkg/text/template/). The template processor can access all _variables_ defined on the command line using the `--vars` argument.

For example to use the `--vars` argument with a playbook step:

```json
{

  "type": "CUSTOM_JAR",

  "name": "Combine Months",

  "actionOnFailure": "CANCEL_AND_WAIT",

  "jar": "s3://snowplow-hosted-assets/3-enrich/hadoop-event-recovery/snowplow-hadoop-event-recovery-0.2.0.jar",

  "arguments": [

    "com.snowplowanalytics.hadoop.scalding.SnowplowEventRecoveryJob",

    "--hdfs",

    "--input",

    "hdfs:///local/monthly/{{.inputVariable}}",

    "--output",

    "hdfs:///local/recovery/{{.outputVariable}}"

  ]

}
```

You would then pass the following command:

```bash
host> ./dataflow-runner run --emr-playbook ${emr-playbook-path} --emr-cluster j-2DPBXD87LSGP9 --vars inputVariable,input,outputVariable,output
```

This would resolve to:

```json
{

  "type": "CUSTOM_JAR",

  "name": "Combine Months",

  "actionOnFailure": "CANCEL_AND_WAIT",

  "jar": "s3://snowplow-hosted-assets/3-enrich/hadoop-event-recovery/snowplow-hadoop-event-recovery-0.2.0.jar",

  "arguments": [

    "com.snowplowanalytics.hadoop.scalding.SnowplowEventRecoveryJob",

    "--hdfs",

    "--input",

    "hdfs:///local/monthly/input",

    "--output",

    "hdfs:///local/recovery/output"

  ]

}
```

The following custom functions are also supported:

- `nowWithFormat [timeFormat]`: where `timeFormat` is a valid Golang [time format](http://golang.org/pkg/time/#Time.Format)
- `timeWithFormat [epoch] [timeFormat]`: where `epoch` is the number of seconds elapsed between January 1st 1970 and a certain point in time as a string and `timeFormat` is valid Golang [time format](http://golang.org/pkg/time/#Time.Format)
- `systemEnv "ENV_VAR"`: where `ENV_VAR` is a key for a valid environment variable
- `base64 [string]`: will base64-encode the string passed as argument
- `base64File "path/to/file.txt"`: will base64-encode the content of the file located at the path passed as argument

### CLI Commands

There are several commands that can be used to manage your data-flow fabric.

#### `up`: Launches a new EMR cluster

```text
NAME:

   dataflow-runner up - Launches a new EMR cluster



USAGE:

   dataflow-runner up [command options] [arguments...]



OPTIONS:

   --emr-config value  EMR config path

   --vars value        Variables that will be used by the templater
```

This command will launch a new cluster ready for step execution, the output should look something like the following:

```text
NAME:

   dataflow-runner run - Adds jobflow steps to a running EMR cluster



USAGE:

   dataflow-runner run [command options] [arguments...]



OPTIONS:

   --emr-playbook value  Playbook path

   --emr-cluster value   Jobflow ID

   --async               Asynchronous execution of the jobflow steps

   --lock value          Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag.

   --softLock value      Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag. Released no matter if the operation failed or succeeded.

   --consul value        Address of the Consul server used for distributed locking

   --vars value          Variables that will be used by the templater
```

#### `run`: Adds jobflow steps to a running EMR cluster

```bash
host> ./dataflow-runner up --emr-config ${emr-config-path}

INFO[0001] Launching EMR cluster with name 'dataflow-runner - sample name'...

INFO[0001] EMR cluster is in state STARTING - need state WAITING, checking again in 20 seconds...

INFO[0021] EMR cluster is in state STARTING - need state WAITING, checking again in 20 seconds...

# this goes for a few lines, omitted for brevity

INFO[0227] EMR cluster is in state STARTING - need state WAITING, checking again in 20 seconds...

INFO[0248] EMR cluster is in state BOOTSTRAPPING - need state WAITING, checking again in 20 seconds...

INFO[0269] EMR cluster is in state BOOTSTRAPPING - need state WAITING, checking again in 20 seconds...

INFO[0289] EMR cluster is in state BOOTSTRAPPING - need state WAITING, checking again in 20 seconds...

INFO[0310] EMR cluster launched successfully; Jobflow ID: j-2DPBXD87LSGP9
```

If the cluster fails to launch (for example, a bootstrap action error or a VPC/quota validation error), Dataflow Runner surfaces the EMR `StateChangeReason` so the reason is visible in the runner's stdout rather than only in the AWS console. From `0.7.7` onwards the output includes both the code and message:

```text
ERRO[0310] EMR cluster state change reason: code='BOOTSTRAP_FAILURE' message="Master instance (i-0123456789abcdef0) failed attempting to download bootstrap action 1 file from S3"

ERRO[0310] Bootstrap failure detected, retrying in 183 seconds...
```

The same code and message are included in the final error returned by the `up` command when all retries are exhausted.

This command adds new steps to the already running cluster. By default this command is blocking - however if you wish to submit and forget simply supply the `--async` argument, the output should look something like the following:

```bash
host> ./dataflow-runner run --emr-playbook ${emr-playbook-path} --emr-cluster j-2DPBXD87LSGP9

INFO[0310] Successfully added 2 steps to the EMR cluster with jobflow id 'j-2DPBXD87LSGP9'...

ERRO[0357] Step 'Combine Months' with id 's-9WZ0VFKC770J' was FAILED

ERRO[0358] Step 'Combine Months 2' with id 's-37F9PKSXBHDAU' was CANCELLED

ERRO[0358] 2/2 steps failed to complete successfully
```

In this case the first step failed which meant that the second step was cancelled. This behavior is dependent on your `actionOnFailure` - you can choose either to:

1. “CANCEL\_AND\_WAIT”: This will cancel all other currently queued jobs and return the cluster to a waiting state ready for new job submissions.
2. “CONTINUE”: This will go to the next step regardless if it failed or not.

**Note**: We have removed the ability to terminate the job flow on failure, to terminate you will need to use the `down` command.

Additionally, Dataflow Runner can acquire a lock before starting the job which can prevent other jobs from running at the same time. Its release will happen when:

- the job has terminated (whether successfully or with failure) with the `--softLock` flag
- the job has succeeded with the `--lock` flag (“hard lock”)

As the above implies, if a job were to fail and the `--lock` flag was used, manual cleaning of the lock will be required.

Additionally, supplying a [Consul](https://www.consul.io/) address, through the `--consul` flag will make this lock distributed.

When the `--consul` flag is used, the lock will be materialized by a key-value pair in Consul for which the key is the value supplied with the `--lock` or `--softLock` argument. Otherwise, it will be materialized by a file on the machine located at the specified path (either relative to your working directory or absolute).

#### `down`: Terminates a running EMR cluster

```text
NAME:

   dataflow-runner down - Terminates a running EMR cluster



USAGE:

   dataflow-runner down [command options] [arguments...]



OPTIONS:

   --emr-config value   EMR config path

   --emr-cluster value  Jobflow ID

   --vars value         Variables that will be used by the templater
```

When you are done with the EMR cluster you can terminate it by using the `down` command. This takes the original emr configuration and the job flow id to then go and terminate the cluster, the output should look something like the following:

```bash
host> ./dataflow-runner down --emr-config ${emr-config-path} --emr-cluster j-2DPBXD87LSGP9

INFO[0358] Terminating EMR cluster with jobflow id 'j-2DPBXD87LSGP9'...

INFO[0358] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...

INFO[0378] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...

INFO[0399] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...

INFO[0420] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...

INFO[0440] Transient EMR run completed successfully
```

#### `run-transient`: Launches, runs and then terminates an EMR cluster

```text
NAME:

   dataflow-runner run-transient - Launches, runs and then terminates an EMR cluster



USAGE:

   dataflow-runner run-transient [command options] [arguments...]



OPTIONS:

   --emr-config value    EMR config path

   --emr-playbook value  Playbook path

   --lock value          Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag.

   --softLock value      Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag. Released no matter if the operation failed or succeeded.

   --consul value        Address of the Consul server used for distributed locking

   --vars value          Variables that will be used by the templater
```

This command is a combination of `up`, `run` and `down` which is designed to mimic the current `EmrEtlRunner` behavior.

As with the `up` command, if the transient cluster fails to launch (for example, a bootstrap action error), Dataflow Runner surfaces the EMR `StateChangeReason` so the reason is visible in the runner's stdout rather than only in the AWS console. From `0.7.8` onwards the code and message are included in both the logs and the final error returned by `run-transient`:

```text
ERRO[0310] EMR cluster state change reason: code='BOOTSTRAP_FAILURE' message="Master instance (i-0123456789abcdef0) failed attempting to download bootstrap action 1 file from S3"
```

---

# Enrich configuration reference
> Complete HOCON configuration reference for Snowplow Enrich applications including monitoring, validation, and stream-specific settings.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/configuration-reference/

This page lists the configuration options for Enrich applications.

## License

Enrich is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run Enrich, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```json
"license": {

  "accept": true

}
```

## Common parameters

| parameter                                                 | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpuParallelismFraction` (since _6.0.0_)                  | Optional. Default: `1`. Controls how the app splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and cpuParallelismFactor = 0.75, then we process 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                                                                         |
| `sinkParallelismFraction` (since _6.0.0_)                 | Optional. Default: `2`. Controls number of sink job that can be run in parallel. E.g. If there are 4 available processors, and sinkParallelismFraction = 2, then we run 8 sink job concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                                                                                                                  |
| `assetsUpdatePeriod`                                      | Optional. E.g. `7 days`. Period after which enrich assets (e.g. the maxmind database for the IpLookups enrichment) should be checked for updates. Assets will never be updated if this key is missing.                                                                                                                                                                                                                                                                             |
| `monitoring.sentry.dsn`                                   | Optional. E.g. `http://sentry.acme.com`. To track uncaught runtime exceptions in Sentry.                                                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.environment` (since _6.7.0_)           | Optional. Environment name to use when reporting exceptions in Sentry.                                                                                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.sentry.tags.*`                                | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.hostname`                      | Optional. E.g. `localhost`. Hostname of the StatsD server to send enrichment metrics (latency and event counts) to.                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.port`                          | Optional. E.g. `8125`. Port of the StatsD server.                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.period`                        | Optional. E.g. `10 seconds`. How frequently to send metrics to StatsD server.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.statsd.tags`                          | Optional. E.g. `{ "env": "prod" }`. Key-value pairs attached to each metric sent to StatsD to provide contextual information.                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.statsd.prefix`                        | Optional. Default: `snowplow.enrich`. Prefix of StatsD metric names.                                                                                                                                                                                                                                                                                                                                                                                                               |
| `monitoring.metrics.prometheus.tags` (since _6.10.0_)     | Optional. E.g. `{ "env": "prod" }`. Key-value pairs added as common labels to all Prometheus metrics. Metrics are exposed at `/metrics` on the health probe port.                                                                                                                                                                                                                                                                                                                  |
| `monitoring.healthProbe.port` (since _6.0.0_)             | Optional. Default: `8000`. Open a HTTP server that returns OK only if the app is healthy.                                                                                                                                                                                                                                                                                                                                                                                          |
| `monitoring.healthProbe.unhealthyLatency` (since _6.0.0_) | Optional. Default: `2 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time.                                                                                                                                                                                                                                                                                                                                         |
| `telemetry.disable`                                       | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                                                                          |
| `telemetry.userProvidedId`                                | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                                                                |
| `validation.acceptInvalid` (since _6.0.0_)                | Optional. Default: `false`. Enrich _6.0.0_ introduces the validation of the enriched events against atomic schema before emitting. If set to `false`, a failed event will be emitted instead of the enriched event if validation fails. If set to `true`, invalid enriched events will be emitted, as before.                                                                                                                                                                      |
| `validation.atomicFieldsLimits` (since _4.0.0_)           | Optional. For the defaults, see [here](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/resources/reference.conf). Configuration for custom maximum atomic fields (strings) length. It's a map-like structure with keys being atomic field names and values being their max allowed length.                                                                                                                                                                  |
| `validation.maxJsonDepth` (since _6.0.0_)                 | Optional. Default: `40`. Maximum allowed depth for the JSON entities in the events. Event will be sent to bad row stream if it contains JSON entity with a depth that exceeds this value.                                                                                                                                                                                                                                                                                          |
| `validation.exitOnJsCompileError` (since _6.0.0_)         | Optional. Default: `true`. If it is set to true, Enrich will exit with error if JS enrichment script is invalid. If it is set to false, Enrich will continue to run if JS enrichment script is invalid but every event will end up as bad row.                                                                                                                                                                                                                                     |
| `decompression.maxBytesInBatch` (since _6.1.0_)           | Optional. Default: `10000000` (10MB). Although a compressed message from the Collector is limited to 1MB, it could become several times bigger after decompression. To avoid loading an enormous amount of data into memory, Enrich will decompress the message in portions (batches). This parameter specifies the maximum size of such a batch. As soon as the decompressed batch reaches `maxBytesInBatch`, it is a emitted for further processing, and a new batch is started. |
| `decompression.maxBytesSinglePayload` (since _6.1.0_)     | Optional. Default: `10000000` (10 MB). Each compressed Collector message contains a number of payloads, which contain one or more events. While the Collector already enforces some payload size limits, this setting exists as a safety check to prevent Enrich from loading large amounts of data into memory. Specifically, if an individual payload exceeds `maxBytesSinglePayload`, it will result in a [size violation](/docs/api-reference/failed-events/#size-violation).  |
| `http.client.requestTimeout` (since _6.5.0_)              | Optional. Default: `5 seconds`. Timeout for internal HTTP requests used by Iglu resolver, alerts, telemetry, and metadata endpoints.                                                                                                                                                                                                                                                                                                                                               |
| `iglu.maxRetry` (since _6.5.0_)                           | Optional. Default: `2`. Maximum number of retries for failed Iglu requests. Lower values allow Enrich to fail faster when Iglu Server is unavailable, falling back to cached schemas instead of blocking on retries.                                                                                                                                                                                                                                                               |
| `iglu.maxWait` (since _6.5.0_)                            | Optional. Default: `1 second`. Maximum wait time for exponential backoff between Iglu request retries.                                                                                                                                                                                                                                                                                                                                                                             |
| `jsAllowedJavaClasses` (since _6.6.0_)                    | Optional. Default: `["*"]`. List of Java classes that the [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/) is allowed to access. This affects both `new <class>` and `<class>.<static method>` usage. Examples: `["java.lang.String", "java.net.URL"]`, `["java.net.*"]`. By default, all classes are allowed.                                                                                                              |
| `compression.enabled` (since _6.11.0_)                    | Optional. Default: `false`. Enable compression on the enriched output stream. Snowplow consumers that support decompression: BigQuery Loader (since _2.2.0_), Databricks Streaming Loader (since _0.5.0_), Elasticsearch Loader (since _3.0.0_), Lake Loader (since _0.10.0_), S3 Loader (since _3.2.0_), Snowflake Loader (since _0.6.0_), Snowbridge (since _5.2.0_). Do not enable this setting if your downstream consumer is not in this list.                                |
| `compression.type` (since _6.11.0_)                       | Optional. Default: `zstd`. Compression algorithm to use. Valid values: `zstd`, `gzip`.                                                                                                                                                                                                                                                                                                                                                                                             |
| `compression.zstdCompressionLevel` (since _6.11.0_)       | Optional. Default: `9`. The `zstd` compression level. Valid range: 1–22. Lower levels compress faster but achieve a worse compression ratio.                                                                                                                                                                                                                                                                                                                                       |
| `compression.gzipCompressionLevel` (since _6.11.0_)       | Optional. Default: `6`. The `gzip` compression level. Valid range: 1–9. Lower levels compress faster but achieve a worse compression ratio.                                                                                                                                                                                                                                                                                                                                        |

## enrich-pubsub

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.pubsub.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.pubsub.reference.hocon).

| parameter                                                        | description                                                                                                                                                                                                                                                                                                                                                      |
| ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                                             | Required. E.g. `projects/example-project/subscriptions/collectorPayloads`. PubSub subscription identifier for the collector payloads.                                                                                                                                                                                                                            |
| `input.durationPerAckExtension`                                  | Optional. Default: `15 seconds`. Pubsub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                                |
| `input.minRemainingAckDeadline`                                  | Optional. Default: `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `60 seconds` and `minRemainingAckDeadline` is `0.1` then the Source will wait until there is `6 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.retries.transientErrors.delay` (since _6.2.0_)            | Optional. Default: `100 millis`. Backoff delay for follow-up attempts of transient GRPC failures retries.                                                                                                                                                                                                                                                        |
| `input.retries.transientErrors.attempts` (since _6.2.0_)         | Optional. Default: `10`. Max number of attempts for transient GRPC failures retries.                                                                                                                                                                                                                                                                             |
| `output.good.topic`                                              | Required. E.g. `projects/example-project/topics/enriched`. Name of the PubSub topic that will receive the enriched events.                                                                                                                                                                                                                                       |
| `output.good.attributes`                                         | Optional. Enriched event fields to add as PubSub message attributes. For example, if this is `[ "app_id" ]` then the enriched event's `app_id` field will be an attribute of the PubSub message, as well as being a field within the enriched event.                                                                                                             |
| `output.good.batchSize`                                          | Optional. Default: `100`. Enriched events are sent to pubsub in batches not exceeding this size.                                                                                                                                                                                                                                                                 |
| `output.good.requestByteThreshold`                               | Optional. Default: `1000000`. Enriched events are sent to pubsub in batches not exceeding this size number of bytes.                                                                                                                                                                                                                                             |
| `output.good.retries.transientErrors.delay` (since _6.2.0_)      | Same as `input.retries.transientErrors.delay` for good events.                                                                                                                                                                                                                                                                                                   |
| `output.good.retries.transientErrors.attempts` (since _6.2.0_)   | Same as `input.retries.transientErrors.attempts` for good events.                                                                                                                                                                                                                                                                                                |
| `output.failed.topic`                                            | Required. E.g. `projects/example-project/topics/failed`. Name of the PubSub topic that will receive the failed events (same format as the enriched events).                                                                                                                                                                                                      |
| `output.failed.batchSize`                                        | Same as `output.good.batchSize` for failed events.                                                                                                                                                                                                                                                                                                               |
| `output.failed.requestByteThreshold`                             | Same as `output.good.requestByteThreshold` for failed events.                                                                                                                                                                                                                                                                                                    |
| `output.failed.retries.transientErrors.delay` (since _6.2.0_)    | Same as `input.retries.transientErrors.delay` for failed events.                                                                                                                                                                                                                                                                                                 |
| `output.failed.retries.transientErrors.attempts` (since _6.2.0_) | Same as `input.retries.transientErrors.attempts` for failed events.                                                                                                                                                                                                                                                                                              |
| `output.bad.topic`                                               | Required. E.g. `projects/example-project/topics/bad`. Name of the PubSub topic that will receive the failed events in the "bad row" format (JSON).                                                                                                                                                                                                               |
| `output.bad.batchSize`                                           | Same as `output.good.batchSize` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                                                |
| `output.bad.requestByteThreshold`                                | Same as `output.good.requestByteThreshold` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                                     |
| `output.bad.retries.transientErrors.delay` (since _6.2.0_)       | Same as `input.retries.transientErrors.delay` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                                  |
| `output.bad.retries.transientErrors.attempts` (since _6.2.0_)    | Same as `input.retries.transientErrors.attempts` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                               |

## enrich-kinesis

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.kinesis.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.kinesis.reference.hocon).

| parameter                                                           | description                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.appName`                                                     | Optional. Default: `snowplow-enrich-kinesis`. Name of the application which the KCL daemon should assume. A DynamoDB table with this name will be created.                                                                                                                                                                                    |
| `input.streamName`                                                  | Required. E.g. `raw`. Name of the Kinesis stream with the collector payloads to read from.                                                                                                                                                                                                                                                    |
| `input.initialPosition.type`                                        | Optional. Default: `TRIM_HORIZON`. Set the initial position to consume the Kinesis stream. Possible values: `LATEST` (most recent data), `TRIM_HORIZON` (oldest available data), `AT_TIMESTAMP` (start from the record at or after the specified timestamp).                                                                                  |
| `input.initialPosition.timestamp`                                   | Required for `AT_TIMESTAMP`. E.g. `2020-07-17T10:00:00Z`.                                                                                                                                                                                                                                                                                     |
| `input.retrievalMode.type`                                          | Optional. Default: `Polling`. Set the mode for retrieving records. Possible values: `Polling` or `FanOut`.                                                                                                                                                                                                                                    |
| `input.retrievalMode.maxRecords`                                    | Required for `Polling`. Default: `1000`. Maximum size of a batch returned by a call to `getRecords`. Records are checkpointed after a batch has been fully processed, thus the smaller `maxRecords`, the more often records can be checkpointed into DynamoDb, but possibly reducing the throughput.                                          |
| `input.retrievalMode.idleTimeBetweenReads`                          | Optional for `Polling`. Default: `200 millis`. Idle time between `getRecords` requests.                                                                                                                                                                                                                                                       |
| `input.workerIdentifier` (since _6.0.0_)                            | Required. Name of this KCL worker used in the DynamoDB lease table.                                                                                                                                                                                                                                                                           |
| `input.leaseDuration` (since _6.0.0_)                               | Optional. Default: `10 seconds`. Duration of shard leases. KCL workers must periodically refresh leases in the DynamoDB table before this duration expires.                                                                                                                                                                                   |
| `input.maxLeasesToStealAtOneTimeFactor` (since _6.0.0_)             | Optional. Default: `2.0`. Controls how to pick the max number of leases to steal at one time. E.g. If there are 4 available processors, and maxLeasesToStealAtOneTimeFactor = 2.0, then allow the KCL to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency.                  |
| `input.checkpointThrottledBackoffPolicy.minBackoff` (since _6.0.0_) | Optional. Default: `100 millis`. Minimum backoff before retrying when DynamoDB provisioned throughput exceeded.                                                                                                                                                                                                                               |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` (since _6.0.0_) | Optional. Default: `1 second`. Maximum backoff before retrying when DynamoDB provisioned throughput limit exceeded.                                                                                                                                                                                                                           |
| `input.debounceCheckpoints` (since _6.0.0_)                         | Optional. Default: `10 seconds`. How frequently to checkpoint our progress to the DynamoDB table. By increasing this value, we can decrease the write-throughput requirements of the DynamoDB table.                                                                                                                                          |
| `input.apiCallAttemptTimeout` (since _6.6.0_)                       | Optional. Default: `15 seconds`. Timeout for API call attempts to Kinesis, DynamoDB, and CloudWatch.                                                                                                                                                                                                                                          |
| `output.good.streamName`                                            | Required. E.g. `enriched`. Name of the Kinesis stream to write to the enriched events.                                                                                                                                                                                                                                                        |
| `output.good.partitionKey`                                          | Optional. How the output stream will be partitioned in Kinesis. Events with the same partition key value will go to the same shard. Possible values: `event_id`, `event_fingerprint`, `domain_userid`, `network_userid`, `user_ipaddress`, `domain_sessionid`, `user_fingerprint`. If not specified, the partition key will be a random UUID. |
| `output.good.throttledBackoffPolicy.minBackoff` (since _6.0.0_)     | Optional. Default: `100 milliseconds`. Minimum backoff before retrying when writing fails with exceeded kinesis write throughput.                                                                                                                                                                                                             |
| `output.good.throttledBackoffPolicy.maxBackoff` (since _6.0.0_)     | Optional. Default: `1 second`. Maximum backoff before retrying when writing fails with exceeded kinesis write throughput.                                                                                                                                                                                                                     |
| `output.good.recordLimit`                                           | Optional. Default: `500`. Maximum allowed to records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                               |
| `output.good.byteLimit`                                             | Optional. Default: `5242880`. Maximum allowed to bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                             |
| `output.good.maxRetries` (since _6.3.0_)                            | Optional. Default: `10`. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                         |
| `output.failed.streamName`                                          | Required. E.g. `failed`. Name of the Kinesis stream that will receive the failed events (same format as the enriched events).                                                                                                                                                                                                                 |
| `output.failed.throttledBackoffPolicy.minBackoff` (since _6.0.0_)   | Same as `output.good.throttledBackoffPolicy.minBackoff` for failed events.                                                                                                                                                                                                                                                                    |
| `output.failed.throttledBackoffPolicy.maxBackoff` (since _6.0.0_)   | Same as `output.good.throttledBackoffPolicy.maxBackoff` for failed events.                                                                                                                                                                                                                                                                    |
| `output.failed.recordLimit`                                         | Same as `output.good.recordLimit` for failed events.                                                                                                                                                                                                                                                                                          |
| `output.failed.byteLimit`                                           | Same as `output.good.byteLimit` for failed events.                                                                                                                                                                                                                                                                                            |
| `output.failed.maxRetries` (since _6.3.0_)                          | Same as `output.good.maxRetries` for failed events.                                                                                                                                                                                                                                                                                           |
| `output.bad.streamName`                                             | Required. E.g. `bad`. Name of the Kinesis stream that will receive the failed events in the "bad row" format (JSON).                                                                                                                                                                                                                          |
| `output.bad.throttledBackoffPolicy.minBackoff` (since _6.0.0_)      | Same as `output.good.throttledBackoffPolicy.minBackoff` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                     |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since _6.0.0_)      | Same as `output.good.throttledBackoffPolicy.maxBackoff` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                     |
| `output.bad.recordLimit`                                            | Same as `output.good.recordLimit` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                           |
| `output.bad.byteLimit`                                              | Same as `output.good.byteLimit` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                             |
| `output.bad.maxRetries` (since _6.3.0_)                             | Same as `output.good.maxRetries` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                            |

## enrich-kafka

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.kafka.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.kafka.reference.hocon).

| parameter                                       | description                                                                                                                                                                               |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                               | Required. Name of the Kafka topic to read collector payloads from.                                                                                                                        |
| `input.bootstrapServers`                        | Required. A list of `host:port` pairs to use for establishing the initial connection to the Kafka cluster                                                                                 |
| `input.debounceCommitOffsets` (since _6.0.0_)   | Optional. Default: `10 seconds`. How frequently to commit our progress back to kafka. By increasing this value, we decrease the number of requests made to the kafka broker.              |
| `input.commitTimeout` (since _6.3.0_)           | Optional. Default: `15 seconds`. The time to wait for offset commits to complete. If an offset commit doesn't complete within this time, a CommitTimeoutException will be raised instead. |
| `input.consumerConf`                            | Optional. Kafka consumer configuration. See [the docs](https://kafka.apache.org/documentation/#consumerconfigs) for all properties.                                                       |
| `output.good.topicName`                         | Required. Name of the Kafka topic to write to                                                                                                                                             |
| `output.good.bootstrapServers`                  | Required. A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                                                                   |
| `output.good.producerConf`                      | Optional. Kafka producer configuration. See [the docs](https://kafka.apache.org/documentation/#producerconfigs) for all properties                                                        |
| `output.good.partitionKey`                      | Optional. Enriched event field to use as Kafka partition key                                                                                                                              |
| `output.good.attributes`                        | Optional. Enriched event fields to add as Kafka record headers                                                                                                                            |
| `output.failed.topicName`                       | Optional. Name of the Kafka topic that will receive the failed events (same format as the enriched events)                                                                                |
| `output.failed.bootstrapServers`                | Same as `output.good.bootstrapServers` for failed events.                                                                                                                                 |
| `output.failed.producerConf`                    | Same as `output.good.producerConf` for failed events.                                                                                                                                     |
| `output.bad.topicName`                          | Optional. Name of the Kafka topic that will receive the failed events in the “bad row” format (JSON)                                                                                      |
| `output.bad.bootstrapServers`                   | Same as `output.good.bootstrapServers` for failed events in the "bad row" format (JSON).                                                                                                  |
| `output.bad.producerConf`                       | Same as `output.good.producerConf` for failed events in the "bad row" format (JSON).                                                                                                      |
| `blobClients.accounts` (since _6.0.0_)          | Optional. Array of Azure Blob Storage accounts to download enrichment assets.                                                                                                             |
| Example values for the Azure storage accounts : |                                                                                                                                                                                           |

- `{ "name": "storageAccount1"}`: public account with no auth
- `{ "name": "storageAccount2", "auth": { "type": "default"} }`: private account using default auth chain
- `{ "name": "storageAccount3", "auth": { "type": "sas", "value": "tokenValue"}}`: private account using SAS token auth

## enrich-nsq

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.nsq.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.nsq.reference.hocon).

| parameter                              | description                                                                                                   |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `input.topic`                          | Required. Name of the NSQ topic with the collector payloads.                                                  |
| `input.lookupHost`                     | Required. The host name of NSQ lookup application.                                                            |
| `input.lookupPort`                     | Required. The port number of NSQ lookup application.                                                          |
| `input.channel`                        | Optional. Default: `collector-payloads-channel`. Name of the NSQ channel used to retrieve collector payloads. |
| `output.good.topic`                    | Required. Name of the NSQ topic that will receive the enriched events.                                        |
| `output.good.nsqdHost`                 | Required. The host name of nsqd application.                                                                  |
| `output.good.nsqdPort`                 | Required. The port number of nsqd application.                                                                |
| `output.failed.topic`                  | Required. Name of the NSQ topic that will receive the failed events (same format as the enriched events).     |
| `output.failed.nsqdHost`               | Required. The host name of nsqd application.                                                                  |
| `output.failed.nsqdPort`               | Required. The port number of nsqd application.                                                                |
| `output.bad.topic`                     | Required. Name of the NSQ topic that will receive the failed events in the "bad row" format (JSON).           |
| `output.bad.nsqdHost`                  | Required. The host name of nsqd application.                                                                  |
| `output.bad.nsqdPort`                  | Required. The port number of nsqd application.                                                                |
| `blobClients.accounts` (since _6.0.0_) | Optional. Array of Azure Blob Storage accounts to download enrichment assets.                                 |

## Enriched events validation against atomic schema

Enriched events are expected to match [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) schema. However, until `3.0.0`, it was never checked that the enriched events emitted by enrich were valid. If an event is not valid against `atomic` schema, a [failed event](/docs/fundamentals/failed-events/) should be emitted instead of the enriched event. However, this is a breaking change, and we want to give some time to users to adapt, in case today they are working downstream with enriched events that are not valid against `atomic`. For this reason, this new validation was added as a feature that can be deactivated like that:

```json
"validation": {

  "acceptInvalid": true

}
```

In this case, enriched events that are not valid against `atomic` schema will still be emitted as before, so that Enrich `3.0.0` can be fully backward compatible. It will be possible to know if the new validation would have had an impact by 2 ways:

1. A new metric `invalid_enriched` has been introduced. It reports the number of enriched events that were not valid against `atomic` schema. As the other metrics, it can be seen on stdout and/or StatsD.
2. Each time there is an enriched event invalid against `atomic` schema, a line will be logged with the failed event (add `-Dorg.slf4j.simpleLogger.log.InvalidEnriched=debug` to the `JAVA_OPTS` to see it).

If `acceptInvalid` is set to `false`, a failed event will be emitted instead of the enriched event in case it's not valid against `atomic` schema.

When we'll know that all our customers don't have any invalid enriched events any more, we'll remove the feature flags and it will be impossible to emit invalid enriched events.

Since `4.0.0`, it is possible to configure the lengths of the atomic fields, below is an example:

```hcl
{

  ...

  # Optional. Configuration section for various validation-oriented settings.

  "validation": {

    # Optional. Configuration for custom maximum atomic fields (strings) length.

    # Map-like structure with keys being field names and values being their max allowed length

    "atomicFieldsLimits": {

        "app_id": 5

        "mkt_clickid": 100000

        # ...and any other 'atomic' field with custom limit

    }

  }

}
```

## Enrichments

The list of the enrichments that can be configured can be found on [this page](/docs/pipeline/enrichments/available-enrichments/).

---

# Enrich Kafka for Azure deployments
> Standalone JVM application for enriching Snowplow events from Kafka topics with configurable enrichments and validation for Azure deployments.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-kafka/

`enrich-kafka` is a standalone JVM application that reads from and writes to Kafka. It can be run from anywhere, as long as it can communicate with your Kafka cluster.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \

-it --rm \

-v $PWD:/snowplow \

snowplow/snowplow-enrich-kafka:6.11.0 \

--enrichments /snowplow/enrichments \

--iglu-config /snowplow/resolver.json \

--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

1. `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
2. Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
3. [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above (for Iglu and enrichments starting from `3.7.0` only).

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich Kafka (since version 3.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich Kinesis for AWS streams
> Standalone JVM application for enriching Snowplow events from AWS Kinesis streams with configurable enrichments and validation.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-kinesis/

`enrich-kinesis` is a standalone JVM application that reads from and writes to Kinesis streams. It can be run from anywhere, as long as it has permissions to access the streams.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \

-it --rm \

-v $PWD:/snowplow \

-e AWS_ACCESS_KEY_ID=xxx \

-e AWS_SECRET_ACCESS_KEY=xxx \

snowplow/snowplow-enrich-kinesis:6.11.0 \

--enrichments /snowplow/enrichments \

--iglu-config /snowplow/resolver.json \

--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

- `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
- Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
- [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above (for Iglu and enrichments starting from `3.7.0` only).

Depending on where the app runs, `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` might not be required.

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich Kinesis (since version 3.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich NSQ for cloud-agnostic applications
> Cloud-agnostic standalone JVM application for enriching Snowplow events from NSQ with configurable enrichments and validation.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-nsq/

`enrich-nsq` is a standalone JVM application that reads from and writes to NSQ. It can be run from anywhere, as long as it can communicate with your NSQ cluster.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \

-it --rm \

-v $PWD:/snowplow \

snowplow/snowplow-enrich-nsq:6.11.0 \

--enrichments /snowplow/enrichments \

--iglu-config /snowplow/resolver.json \

--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

1. `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
2. Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
3. [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above.

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich NSQ (since version 3.8.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich PubSub for Google Cloud
> Standalone JVM application for enriching Snowplow events from Google Cloud PubSub with configurable enrichments and validation.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-pubsub/

`enrich-pubsub` is a standalone JVM application that reads from and writes to PubSub topics. It can be run from anywhere, as long as it has permissions to access the topics.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \

-it --rm \

-v $PWD:/snowplow \

-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/snowplow-gcp-account-11aa55ff6b1b.json \

snowplow/snowplow-enrich-pubsub:6.11.0 \

--enrichments /snowplow/enrichments \

--iglu-config /snowplow/resolver.json \

--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

1. GCP credentials [JSON file](https://cloud.google.com/docs/authentication/getting-started)
2. `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
3. Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
4. enrich-pubSub [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above (for Iglu and enrichments starting from `3.7.0` only).

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich PubSub (since version 3.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich applications for different platforms
> Technical documentation for Snowplow enrichment applications that validate and enrich collector payloads on Kinesis, PubSub, Kafka, and NSQ.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/

This is the technical documentation for the enrichment step. If you are not familiar yet with this step of the pipeline, please refer to [this page](/docs/pipeline/enrichments/).

Here is the list of the enrichment assets:

## [enrich-kinesis](/docs/api-reference/enrichment-components/enrich-kinesis/) (AWS)

Standalone JVM application that reads collector payloads events from a Kinesis stream and outputs back to Kinesis.

## [enrich-pubsub](/docs/api-reference/enrichment-components/enrich-pubsub/) (GCP)

Standalone JVM application that reads collector payloads from a PubSub subscription and outputs back to PubSub.

## [enrich-kafka](/docs/api-reference/enrichment-components/enrich-kafka/) (Azure)

Standalone JVM application that reads collector payloads from a Kafka topic and outputs back to Kafka.

## [enrich-nsq](/docs/api-reference/enrichment-components/enrich-nsq/) (cloud agnostic)

Standalone JVM application that reads collector payloads from NSQ and outputs back to NSQ.

---

# Monitoring in Enrich applications
> Monitor Snowplow Enrich applications with StatsD or Prometheus metrics for event counts and latency, plus Sentry error reporting.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/monitoring/

Enrich has monitoring built in, to help the pipeline operator.

## Metrics

Enrich tracks the following metrics:

- `raw`: total number of raw collector payloads received.
- `good`: total number of events successfully enriched.
- `good_bytes`: total byte size of successfully enriched events, measured after serialization to TSV format.
- `failed` (`incomplete` before version 6.0.0): the number of failed events produced due to [common failures](/docs/fundamentals/failed-events/#common-failures), e.g. schema violations or enrichment failures.
- `bad`: total number of all types of failed events, e.g. due to schema violations, invalid collector payloads, or enrichment failures.
- `dropped`: total number of events explicitly dropped by calling `event.drop()` in a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/#discard-events).
- `e2e_latency_millis` (`latency` before version 6.0.0): time difference between the collector timestamp and when the event is emitted to the output stream.
- `latency_millis` (since version 6.0.0): delay between an input record being written to the stream and Enrich starting to process it.
- `invalid_enriched`: number of enriched events that were not valid against the [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) schema.

The count metrics (`raw`, `good`, `good_bytes`, `bad`, `failed`, `dropped`, and `invalid_enriched`) refer to the count since the previous measurement. A collector payload can carry multiple events, so it is possible for `good` to be larger than `raw`.

The latency metrics (`e2e_latency_millis` and `latency_millis`) refer to the maximum latency across all events since the previous measurement.

You can report these metrics using StatsD or Prometheus (or both).

## StatsD

[StatsD](https://github.com/statsd/statsd) is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a [pluggable storage backend](https://github.com/statsd/statsd/blob/master/docs/backend.md).

Enrich can periodically push its metrics to a StatsD daemon. For example, a single emission looks like:

```text
snowplow.enrich.raw:42|c|#tag1:value1

snowplow.enrich.good:30|c|#tag1:value1

snowplow.enrich.good_bytes:30000|c|#tag1:value1

snowplow.enrich.failed:10|c|#tag1:value1

snowplow.enrich.bad:12|c|#tag1:value1

snowplow.enrich.dropped:0|c|#tag1:value1

snowplow.enrich.e2e_latency_millis:123.4|g|#tag1:value1

snowplow.enrich.latency_millis:123.4|g|#tag1:value1

snowplow.enrich.invalid_enriched:0|c|#tag1:value1
```

StatsD is configured under `monitoring.metrics.statsd` in your configuration file:

```json
"monitoring": {

  "metrics": {

    "statsd": {

      "hostname": "localhost"

      "port": 8125

      "tags": {

        "tag1": "value1"

        "tag2": "value2"

      }

      "prefix": "snowplow.enrich"

      "period": "1 minute"

    }

  }

}
```

## Prometheus

[Prometheus](https://prometheus.io/) is an open-source monitoring system that scrapes metrics from an HTTP endpoint exposed by the application. Enrich exposes its metrics at `/metrics` on the health probe port (default `8000`), in the standard Prometheus text format.

Prometheus is configured under `monitoring.metrics.prometheus` in your configuration file:

```json
"monitoring": {

  "metrics": {

    "prometheus": {

      "tags": {

        "tag1": "value1"

        "tag2": "value2"

      }

    }

  }

}
```

The `tags` map adds common labels to all metrics. Point your Prometheus scrape config at `http://<host>:<port>/metrics`, where `<port>` is the health probe port (default `8000`, configured via `monitoring.healthProbe.port`).

## Sentry

[Sentry](https://docs.sentry.io/) is a popular error monitoring service, which helps developers diagnose and fix problems in an application. Enrich can send an error report to Sentry whenever something unexpected happens when trying to enrich an event. The reasons for the error can then be explored in the Sentry server's UI.

Sentry is configured by setting the `monitoring.sentry.dsn` key in your configuration file with the URL of your Sentry server:

```json
"monitoring": {

  "sentry": {

    "dsn": "http://sentry.acme.com"

  }

}
```

---

# Enrich 4.0.x upgrade guide
> Upgrade guide for Snowplow Enrich 4.0.x covering license acceptance, atomic field limits, and deprecated assets.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/upgrade-guides/4-0-x-upgrade-guide/

## Breaking changes

### New license

Since version 4.0.0, Enrich has been migrated to use the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

### `stream-enrich` assets and `enrich-rabbitmq` deprecated

As announced a while ago, these assets are now removed from the codebase.

## Upgrading

### License acceptance

You have to explicitly accept the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)). To do so, either set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable, or update the following section in the configuration:

```hcl
{

  license {

    accept = true

  }

  ...

}
```

### Atomic fields limits

Several [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) fields, such as `mkt_clickid` have length limits defined (in this case, 128 characters). Recent versions of Enrich enforce these limits, so that oversized data does not break loading into the warehouse columns. However, over time we’ve observed that valid data does not always fit these limits. For example, TikTok click ids can be up to 500 (or 1000, according to some sources) characters long.

In this release, we are adding a way to configure the limits, and we are increasing the default limits for several fields:

- `mkt_clickid` limit increased from `128` to `1000`
- `page_url` limit increased from `4096` to `10000`
- `page_referrer` limit increased from `4096` to `10000`

Depending on your [configuration](/docs/api-reference/enrichment-components/configuration-reference/), this might be a breaking change:

- If you have `featureFlags.acceptInvalid` set to `true` in Enrich, then you probably don’t need to worry, because you had no validation in the first place (although we do recommend to enable it).

- If you have `featureFlags.acceptInvalid` set to `false` (default), then previously invalid events might become valid (which is a good thing), and you need to prepare your warehouse for this eventuality:

  - For Redshift, you should resize the respective columns, e.g. to `VARCHAR(1000)` for `mkt_clickid`. If you don’t, Redshift will truncate the values.
  - For Snowflake and Databricks, we recommend removing the VARCHAR limit altogether. Otherwise, loading might break with longer values. Alternatively, you can alter the Enrich configuration to revert the changes in the defaults.
  - For BigQuery, no steps are necessary.

Below is an example of how to configure these limits:

```hcl
{

  ...

  # Optional. Configuration section for various validation-oriented settings.

  "validation": {

    # Optional. Configuration for custom maximum atomic fields (strings) length.

    # Map-like structure with keys being field names and values being their max allowed length

    "atomicFieldsLimits": {

        "app_id": 5

        "mkt_clickid": 100000

        # ...and any other 'atomic' field with custom limit

    }

  }

}
```

---

# Enrich 6.0.x upgrade guide
> Upgrade guide for Snowplow Enrich 6.0.x covering common-streams refactoring, configuration changes, and deprecated features.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/upgrade-guides/6-0-x-upgrade-guide/

In version 6.0.0, Enrich is refactored to use [common-streams](https://github.com/snowplow-incubator/common-streams) libraries under the hood. [common-streams](https://github.com/snowplow-incubator/common-streams) is the collection of libraries that contains streaming-related constructs commonly used across many Snowplow streaming applications.

[common-streams](https://github.com/snowplow-incubator/common-streams) allows for the adjustment of many different settings. It also provides default values for most of these settings, which are battle-tested. Therefore, we recommend using default values whenever it is possible. You can find more information about defaults in [configuration reference](/docs/api-reference/enrichment-components/configuration-reference/).

Also, we took this opportunity to make a few breaking changes. Here are the changes:

### Config Field Changes

In version 6.0.0, some of the config fields are renamed or moved to a different section. Here are these changes:

- `incomplete` stream config field is renamed to `failed`.

- `acceptInvalid` and `exitOnJsCompileError` fields under `featureFlags` section are moved under the `validation` section.

- `experimental.metadata` section is moved to the root level.

- In enrich-kafka, `output.good.headers` field is renamed to `output.good.attributes`.

- In enrich-kafka, `blobStorage.azureStorage.accounts` section is moved to the `blobClients.accounts`.

- In enrich-kafka, we are now using [static membership](https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances) for the consumer, to reduce rebalancing in case of pod restart or crash. The default value for `group.instance.id` is set to the host name.

### Feature Deprecations

- Output `pii` stream is removed as in our experience it is not used. There will no longer be an option to write `pii_transformation` events to an extra output stream.

- Remote adapters are removed. This was another feature with little to no usage that allowed Enrich to support custom payloads (which would be sent to a configured URL for translation into the expected format). In practice, most of this can be already achieved with [Iglu Webhooks](/docs/sources/webhooks/iglu-webhook/).

- Reading events from files and writing events to files is no longer supported. This has never been a viable option for production setups.

- In enrich-kinesis, passing enrichment configs to the application via DynamoDB is no longer possible.

- In enrich-kinesis, it is no longer possible to send KCL metrics to Cloudwatch.

## Metrics Changes

Existing metrics will continue to be emitted. Three new metrics are added:

- `failed`: Same value as the `incomplete` metric, for transition. The goal is to remove `incomplete` metric and to be consistent with the naming of streams/topics in configuration

- `e2e_latency_millis`: Same value as the `latency` metric, for transition. The goal is to remove `latency` metric so that the naming is consistent across applications

- `latency_millis`: Delay between the input record getting written to the stream and Enrich starting to process it

Furthermore, the old `latency` metric has changed subtly. Before, it represented the latency of the most recently processed event. Now it refers to the _maximum latency of all events_ since the previous metric was emitted.

---

# Upgrade guides for Enrich
> Step-by-step guides for upgrading Snowplow Enrich applications to newer versions with breaking changes and migration instructions.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/upgrade-guides/

This section contains information to help you upgrade to newer versions of Enrich.

## [4.0.x upgrade guide](/docs/api-reference/enrichment-components/upgrade-guides/4-0-x-upgrade-guide/)

[Upgrade guide for Snowplow Enrich 4.0.x covering license acceptance, atomic field limits, and deprecated assets.](/docs/api-reference/enrichment-components/upgrade-guides/4-0-x-upgrade-guide/)

## [6.0.x upgrade guide](/docs/api-reference/enrichment-components/upgrade-guides/6-0-x-upgrade-guide/)

[Upgrade guide for Snowplow Enrich 6.0.x covering common-streams refactoring, configuration changes, and deprecated features.](/docs/api-reference/enrichment-components/upgrade-guides/6-0-x-upgrade-guide/)

---

# Failed event types
> Reference guide for Snowplow failed event types including schema violations, enrichment failures, and loader errors, with recovery recommendations.
> Source: https://docs.snowplow.io/docs/api-reference/failed-events/

This page lists all the possible types of [failed events](/docs/fundamentals/failed-events/).

## Where do failed events originate?

While an event is being processed by the pipeline it is checked to ensure it meets the specific formatting or configuration expectations. These include checks like: does it match the schema it is associated with, were Enrichments successfully applied, and was the payload sent by the tracker acceptable.

Generally, the [Collector](/docs/api-reference/stream-collector/) tries to write any payload to the raw stream, no matter its content, and no matter whether it is valid. This explains why many of the failure types are filtered out by the [Enrich](/docs/api-reference/enrichment-components/) application, and not any earlier.

> **Note:** The Collector might receive events in batches. If something is wrong with the Collector payload as a whole (e.g. due to a [Collector payload format violation](#collector-payload-format-violation)), the generated failed event would represent an entire batch of Snowplow events.
>
> Once the Collector payload successfully reaches the validation and enrichment steps, it is split into its constituent events. Each of them would fail (or not fail) independently (e.g. due to an [enrichment failure](#enrichment-failure)). This means that each failed event generated at this stage represents a single Snowplow event.

## Schema violation

This failure type is produced during the process of [validation and enrichment](/docs/pipeline/enrichments/). It concerns the [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) which can be attached to your snowplow event.

**Details**

In order for an event to be processed successfully:

1. There must be a schema in an [Iglu repository](/docs/api-reference/iglu/iglu-repositories/) corresponding to each self-describing event or entity. The enrichment app must be able to look up the schema in order to validate the event.
2. Each self-describing event or entity must conform to the structure described in the schema. For example, all required fields must be present, and all fields must be of the expected type.

If your pipeline is generating schema violations, it might mean there is a problem with your tracking, or a problem with your [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/) which lists where schemas should be found. The error details in the schema violation JSON object should give you a hint about what the problem might be.

Snowplow customers should check in the Snowplow Console that all data structures are correct and have been [promoted to production](/docs/event-studio/data-structures/). Snowplow Self-Hosted users should check that the Enrichment app is configured with an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) that points to a repository containing the schemas.

Next, check the tracking code in your custom application, and make sure the entities you are sending conform to the schema definition.

Once you have fixed your tracking, you might want to also [recover the failed events](/docs/monitoring/recovering-failed-events/), to avoid any data loss.

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Schema violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema).

## Enrichment failure

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, and it represents any failure to enrich the event by one of your configured enrichments.

**Details**

There are many reasons why an enrichment will fail, but here are some examples:

- You are using the [custom SQL enrichment](/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/) but the credentials for accessing the database are wrong
- You are using the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) but have mis-configured the location of the MaxMind database
- You are using the [custom API request enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/) but the API server is not responding
- The raw event contained an unstructured event field or a context field which was not valid JSON
- An Iglu server responded with an unexpected error response, so the event schema could not be resolved

If your pipeline is generating enrichment failures, it might mean there is a problem with your enrichment configuration. The error details in the enrichment failure JSON object should give you a hint about what the problem might be.

Once you have fixed your enrichment configuration, you might want to also [recover the failed events](/docs/monitoring/recovering-failed-events/), to avoid any data loss.

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Enrichment failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema).

## Collector payload format violation

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, when Collector payloads from the raw stream are deserialized.

**Details**

Violations could be:

- Malformed HTTP requests
- Truncation
- Invalid query string encoding in URL
- Path not respecting `/vendor/version`

The most likely source of this failure type is bot traffic that has hit the Collector with an invalid HTTP request. Bots are prevalent on the web, so do not be surprised if your Collector receives some of this traffic. Generally you would ignore, and not try to recover, a Collector payload format violation, because it likely did not originate from a tracker or a webhook.

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Collector payload format violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/collector_payload_format_violation/jsonschema).

## Adapter failure

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, when it tries to interpret a Collector payload from the raw stream as a HTTP request from a [3rd party webhook](/docs/sources/webhooks/).

> **Info:** Many adapter failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline.

**Details**

The failure could be:

1. The vendor/version combination in the Collector URL is not supported. For example, imagine an HTTP request sent to `/com.sandgrod/v3` which is a mis-spelling of the [sendgrid adapter](http://sendgrid.com) endpoint.
2. The webhook sent by the 3rd party does not conform to the expected structure and list of fields for this webhook. For example, imagine the 3rd party webhook payload is updated and stops sending a field that it was sending before.

Many adapter failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline. However, if you believe you are missing data because of a misconfigured webhook, then you might try to fix the webhook and then [recover the failed events](/docs/monitoring/recovering-failed-events/).

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Adapter failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema).

## Tracker protocol violation

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, when an HTTP request does not conform to our [Snowplow Tracker Protocol](/docs/events/).

**Details**

Snowplow trackers send HTTP requests to the `/i` endpoint or the `/com.snowplowanalytics.snowplow/tp2` endpoint, and they are expected to conform to this protocol.

Many tracker protocol violations are caused by bot traffic, so do not be surprised to see some of them in your pipeline.

Another likely source is misconfigured query parameters if you are using the [pixel tracker](/docs/sources/pixel-tracker/). In this case you might try to fix your application sending events, and then [recover the failed events](/docs/monitoring/recovering-failed-events/).

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Tracker protocol violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema).

## Size violation

This failure type can be produced either by the [Collector](/docs/api-reference/stream-collector/) or by the [Enrich](/docs/pipeline/enrichments/) application. It happens when the size of the raw event or enriched event is too big for the output message queue. In this case it will be truncated and wrapped in a size violation failed event instead.

**Details**

Failures of this type cannot be [recovered](/docs/monitoring/recovering-failed-events/). The best you can do is to fix any application that is sending over-sized events.

Because this failure is handled during collection or enrichment, events in the real time good stream are free of this violation type.

The size violation schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/size_violation/jsonschema/1-0-0).

## Loader parsing error

This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/), if the enriched event in the real time good stream cannot be parsed as a canonical TSV event format. For example, if the row does not have enough columns (131 are expected) or the `event_id` is not a UUID. This error type is uncommon and unexpected, because it can only be caused by an invalid message in the stream of validated enriched events.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

The loader parsing error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_parsing_error/jsonschema/2-0-0).

## Loader Iglu error

This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/) and describes an error using the [Iglu](/docs/api-reference/iglu/) subsystem.

**Details**

For example:

- A schema is not available in any of the repositories listed in the [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/).
- Some loaders (e.g. [RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) and [Postgres loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/)) make use of the "schema list" API endpoints, which are only implemented for an [Iglu server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) repository. A loader Iglu error will be generated if the schema is in a [static repo](/docs/api-reference/iglu/iglu-repositories/static-repo/).
- The loader cannot auto-migrate a database table. If a schema version is incremented from `1-0-0` to `1-0-1` then it is expected to be [a non-breaking change](/docs/fundamentals/schemas/versioning/#schema-version-format), and many loaders (e.g. RDB loader) attempt to execute a `ALTER TABLE` statement to facilitate the new schema in the warehouse. But if the schema change is breaking (e.g. string field changed to integer field) then the database migration is not possible.

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Loader Iglu error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_iglu_error/jsonschema/2-0-0).

## Loader recovery error (legacy)

Only the [BigQuery repeater](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/#snowplow-bigquery-repeater) generated this error. We call it "loader recovery error" because the purpose of the repeater was to recover from previously failed inserts. It represents the case when the software could not re-insert the row into the database due to a runtime failure or invalid data in a source.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Loader recovery error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_recovery_error/jsonschema/1-0-0)

## Loader runtime error

This failure type can be produced by any loader and describes generally any runtime error that we did not catch. For example, a DynamoDB outage, or a null pointer exception. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Loader runtime error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-0-1).

## Relay failure

This failure type is only produced by relay jobs, which transfer Snowplow data into a 3rd party platform. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Relay failure schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/relay_failure/jsonschema/1-0-0).

## Generic error

This is a failure type for anything that does not fit into the other categories, and is unlikely enough that we have not created a special category. The failure error messages should give you a hint about what has happened.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Generic error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/generic_error/jsonschema/1-0-0).

---

# Iglu Server
> Iglu Server is a RESTful schema registry for publishing, validating, and serving Iglu schemas in self-hosted Snowplow pipelines.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/iglu-server/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

[Iglu Server](https://github.com/snowplow/iglu-server) is the schema registry used by self-hosted Snowplow pipelines. It stores your [schemas](/docs/fundamentals/schemas/), validates them on upload, and serves them over a RESTful API.

## What you can do with it

- Publish and update [self-describing schemas](/docs/fundamentals/schemas/), with validation at upload time.
- Serve schemas to your pipeline components via the [Iglu Resolver](/docs/api-reference/iglu/iglu-resolver/).
- Issue scoped API keys, so different teams or applications can read or write only the vendor prefixes they own.
- Receive webhook notifications when schemas are published or updated. See [`webhooks.schemaPublished`](/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/) in the configuration reference.
- Mark schema versions as [superseded](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded) when you need to fix a schema that's already in production.

Most users interact with Iglu Server through [Igluctl](/docs/api-reference/iglu/igluctl/) rather than calling the API directly. See [Manage schemas](/docs/api-reference/iglu/manage-schemas/).

## REST API

Iglu Server exposes a RESTful API for managing schemas, validating data, and issuing API keys. The API is documented interactively via Swagger UI at:

```text
http://YOUR_IGLU_HOST/static/swagger-ui/index.html
```

Iglu Server expects an API key in the `apikey` header for any endpoint that requires read access to private schemas or write access to the registry. See [Generate API keys](#generate-api-keys) below for how to generate them.

Iglu Server is released under the [Snowplow Limited Use License](https://docs.snowplow.io/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)). To accept the terms of the license and run Iglu Server, set the environment variable `ACCEPT_LIMITED_USE_LICENSE=yes`, or set `license.accept = true` in your HOCON config file.

## Deploy Iglu Server

The recommended way to deploy Iglu Server is to use the Terraform modules provided as part of the [Snowplow Self-Hosted quick start](/docs/get-started/self-hosted/quick-start/#set-up-iglu-server). The quick start walks through deploying Iglu Server (along with the rest of your pipeline) on AWS, GCP, or Azure.

The Terraform modules are also available directly on the Terraform Registry if you want to deploy Iglu Server independently:

- [AWS](https://registry.terraform.io/modules/snowplow-devops/iglu-server-ec2/aws/latest)
- [GCP](https://registry.terraform.io/modules/snowplow-devops/iglu-server-ce/google/latest)
- [Azure](https://registry.terraform.io/modules/snowplow-devops/iglu-server-vmss/azurerm/latest)

### Deploy with Docker

If you'd rather run Iglu Server manually, you'll also need a PostgreSQL instance to back it. Iglu Server stores schemas in PostgreSQL and connects to it on startup. The Terraform modules handle this for you; if you're deploying manually, provision your own database.

Pull the Iglu Server image from Docker Hub and pass a HOCON config file:

```bash
docker pull snowplow/iglu-server:0.14.1



docker run --rm \

-v $PWD/config.hocon:/iglu/config.hocon \

-e ACCEPT_LIMITED_USE_LICENSE=yes \

snowplow/iglu-server:0.14.1 --config /iglu/config.hocon
```

A minimal configuration looks like this:

```hocon
{

  "database": {

    "host": "postgres"

    "dbname": "igludb"

    "username": "postgres"

    "password": "mysecret"

  }



  "superApiKey": "bb7b7503-40d3-459c-943a-f8d31a6f5638"

}
```

See the [configuration reference](/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/) for all available parameters. A [docker-compose.yml](https://github.com/snowplow/iglu-server/blob/master/docker/docker-compose.yml) is also provided in the repository to help you get started.

#### Initialize the database

For a fresh install, create the database, then run the `setup` command to create the tables:

```bash
psql -U postgres -c "CREATE DATABASE igludb"
```

```bash
docker run --rm \

-v $PWD/config.hocon:/iglu/config.hocon \

-e ACCEPT_LIMITED_USE_LICENSE=yes \

snowplow/iglu-server:0.14.1 setup --config /iglu/config.hocon
```

## Generate API keys

The super API key in your configuration file lets you generate further keys for your applications via the API key generation service.

To generate a pair of read and write API keys for a specific vendor prefix, send a `POST` request to `HOST/api/auth/keygen` with your super API key in the `apikey` header:

```bash
curl HOST/api/auth/keygen \

  -X POST \

  -H 'apikey: your_super_apikey' \

  -d '{"vendorPrefix":"com.acme"}'
```

You'll receive a JSON response with the two keys:

```json
{

  "read": "an-uuid",

  "write": "another-uuid"

}
```

To revoke a key, send a `DELETE` request to the same endpoint:

```bash
curl 'HOST/api/auth/keygen?key=some-uuid' \

  -X DELETE \

  -H 'apikey: your_super_apikey'
```

## Optional configuration

### Dummy mode for testing

Iglu Server supports a dummy database mode that stores all data in memory rather than PostgreSQL. Use it for testing only — all data is lost on restart.

To enable dummy mode, set `database.type` to `"dummy"` in your config. The dummy server uses a single hardcoded master API key: `48b267d7-cd2b-4f22-bae4-0f002008b5ad`.

### Logging

Iglu Server uses [SLF4J Simple Logger](https://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html), configurable via system properties. For example, to redirect logs to a file and suppress verbose connection loop output:

```bash
-Dorg.slf4j.simpleLogger.logFile=server.log

-Dorg.slf4j.simpleLogger.log.org.http4s.blaze.channel.nio1.SelectorLoop=warn
```

At debug log level, `SchemaService` prints all HTTP requests and responses.

---

# Iglu Server configuration reference
> Complete reference of all configuration options for Iglu Server, including database, networking, webhooks, and advanced settings.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

This is a complete list of the options that can be configured in the Iglu Server HOCON config file. The [examples on GitHub](https://github.com/snowplow/iglu-server/tree/master/config) show how to prepare this file.

## License

Iglu Server is released under the [Snowplow Limited Use License](https://docs.snowplow.io/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run Iglu Server, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

## Common options

| parameter                                          | description                                                                                                                                                                                                                                                                    |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `repoServer.interface`                             | Optional. Default: `0.0.0.0`. Address on which the server listens to http connections.                                                                                                                                                                                         |
| `repoServer.port`                                  | Optional. Default: `8080`. Port on which the server listens.                                                                                                                                                                                                                   |
| `repoServer.idleTimeout`                           | Default: `30 seconds`. TCP connections are dropped after this timeout expires. In case Iglu Server runs behind a load balancer, this should slightly exceed the load balancer's idle timeout.                                                                                  |
| `repoServer.hsts.enable` _(since 0.12.0)_          | Default: `false`. Whether to send an [HSTS header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security).                                                                                                                                       |
| `repoServer.hsts.maxAge` _(since 0.12.0)_          | Default: `365 days`. The maximum age for the HSTS header.                                                                                                                                                                                                                      |
| `repoServer.maxPayloadSize` _(since 0.14.0)_       | Default: `100000`. Maximum accepted request body size in bytes. Requests exceeding this are rejected.                                                                                                                                                                          |
| `database.type`                                    | Optional. Default: `postgres`. Can be changed to `dummy` during development for in-memory only storage.                                                                                                                                                                        |
| `database.host`                                    | Required. Host name for Postgres database.                                                                                                                                                                                                                                     |
| `database.port`                                    | Optional. Default: `5432`. Port for Postgres database.                                                                                                                                                                                                                         |
| `database.dbname`                                  | Required. Name of Postgres database.                                                                                                                                                                                                                                           |
| `database.username`                                | Required. Username for connecting to Postgres.                                                                                                                                                                                                                                 |
| `database.password`                                | Required. Password for connecting to Postgres.                                                                                                                                                                                                                                 |
| `swagger.baseUrl`                                  | Optional. Example: `/custom/prefix`. Customize the api base url in Swagger. Helpful for when running Iglu Server behind a proxy server.                                                                                                                                        |
| `debug`                                            | Optional. Default: `false`. Enable additional debug api endpoint to respond with all internal state.                                                                                                                                                                           |
| `patchesAllowed`                                   | Optional. Default: `false`. If `true`, allows overwriting a given version of a schema with new content. See [amending schemas](/docs/fundamentals/schemas/versioning/).                                                                                                        |
| `webhooks.schemaPublished`                         | Optional. Array with the list of webhooks that will be called when a schema is published or updated with a vendor that matches the specified prefixes. See the [examples on GitHub](https://github.com/snowplow/iglu-server/blob/0.8.7/config/config.reference.hocon#L81-L99). |
| `webhooks.schemaPublished.uri`                     | Required. URI of the HTTP server that will receive the webhook event.                                                                                                                                                                                                          |
| `webhooks.schemaPublished.vendorPrefixes`          | Optional. Example: `["com", "org.acme", "org.snowplow"]`. List of schema prefixes (regexes) that should be sent via the webhook.                                                                                                                                               |
| `webhooks.schemaPublished.usePost` (since _0.8.7_) | Optional. Default: `false`. Whether to use `POST` to send request via the webhook.                                                                                                                                                                                             |
| `superApiKey`                                      | Optional. Set a super api key with permission to read/write any schema, and add other api keys.                                                                                                                                                                                |
| `maxJsonDepth` _(since 0.14.0)_                    | Default: `40`. Maximum nesting depth for JSON values accepted by the server. Used both when validating schemas and when parsing incoming JSON payloads.                                                                                                                        |

## Advanced options

These advanced options have sensible defaults and rarely need changing.

| parameter                                 | description                                                                                                                                                                                                                 |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `repoServer.threadPool.type`              | Default: `fixed` for a fixed thread pool. Can be `cached` for a cached thread pool. Type of the thread pool used by the underlying BlazeServer for executing Futures.                                                       |
| `repoServer.threadPool.size`              | Optional. Default: `4`. Size of the thread pool if the type is `fixed`.                                                                                                                                                     |
| `repoServer.maxConnections`               | Optional. Default: `1024`. Maximum number of client connections that may be active at any time.                                                                                                                             |
| `database.pool.type`                      | Optional. Default: `hikari` (recommended for production). Can be changed to `nopool` to remove the upper bound on the number of connections.                                                                                |
| `database.pool.maximumPoolSize`           | Optional. Default: `5`. Maximum number of connections in the Hikari pool.                                                                                                                                                   |
| `database.pool.connectionTimeout`         | Optional. Default: `30 seconds`. Timeout on the Hikari connection pool.                                                                                                                                                     |
| `database.pool.maxLifetime`               | Optional. Default: `1800 seconds`. Maximum lifetime of a connection in the Hikari pool.                                                                                                                                     |
| `database.pool.minimumIdle`               | Optional. Default: `5`. Minimum number of idle connections in the Hikari pool.                                                                                                                                              |
| `database.pool.connectionPool.type`       | Optional. Default: `fixed` for a fixed thread pool (recommended in production). Type of the thread pool used for awaiting connection to the database.                                                                       |
| `database.pool.connectionPool.size`       | Optional. Default: `4`. Number of threads to use when the connection pool has type `fixed`.                                                                                                                                 |
| `database.pool.transactionPool.type`      | Optional. Default: `cached` (recommended for production). Type of the thread pool used for blocking JDBC operations.                                                                                                        |
| `preTerminationPeriod` (since _0.8.0_)    | Optional. Default: `1 second`. How long the server should pause after receiving a sigterm before starting the graceful shutdown. During this period the server continues to accept new connections and respond to requests. |
| `preTerminationUnhealthy` (since _0.8.0_) | Optional. Default: `false`. During the `preTerminationPeriod`, the server can be configured to return 503s on the `/health` endpoint. Can be helpful for removing the server from a load balancer's targets.                |

---

# Introduction to Iglu repositories for schema storage
> Remote Iglu repositories for storing and serving JSON schemas via HTTP.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

An Iglu repository acts as a store of JSON schemas. Hosting JSON schemas in an Iglu repository allows you to use those schemas in Snowplow.

There are two Iglu repository technologies available for deploying your own Iglu repository:

| **Repository**                                                               | **Description**                                          |
| ---------------------------------------------------------------------------- | -------------------------------------------------------- |
| [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/)       | An Iglu repository server structured as a RESTful API    |
| [Static repository](/docs/api-reference/iglu/iglu-repositories/static-repo/) | An Iglu repository server structured as a static website |

## Iglu Central

[Iglu Central](https://iglucentral.com/) is a public, machine-readable repository of JSON schemas hosted by Snowplow. It contains the schemas for Snowplow's out-of-the-box events and entities.

Under the hood, Iglu Central is a [static Iglu repository](/docs/api-reference/iglu/iglu-repositories/static-repo/) — a schema registry served as a static website over HTTP. The root index page at [iglucentral.com](https://iglucentral.com/) links to all the schemas it hosts.

---

# Static Iglu repository
> Host Iglu schemas on any HTTP server as a read-only alternative to Iglu Server.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/static-repo/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

A static Iglu repository is a read-only schema registry served from a static website. It's a lighter alternative to [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) when you don't need an authenticated write API. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is itself a static repo.

You can host a static repo on any HTTP server that can serve files with a directory structure — for example, Nginx, Apache, Amazon S3, Google Cloud Storage, or Azure Blob Storage.

> **Warning:** Some loaders (notably [RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)) use the schema-list API endpoints that are only fully supported by Iglu Server. If you use a static repo with these loaders, you must generate the schema-list objects when you publish. [`igluctl`](/docs/api-reference/iglu/igluctl/) generates them by default; if you upload schemas to your host by hand, the lists won't exist and the loader will fail.

## Schema file structure

Iglu repositories use a fixed folder hierarchy keyed by vendor, schema name, format, and version:

```text
schemas

└── com.acme

    └── ad_click

        └── jsonschema

            └── 1-0-0
```

The version file (`1-0-0`) has no extension and contains the self-describing JSON schema. See [Self-describing schemas](/docs/fundamentals/schemas/#self-describing-json-schema-anatomy) for what goes inside.

## Publish schemas

Prepare your schemas locally in the folder structure above. Then publish them to your HTTP host.

If you're hosting on Amazon S3, [`igluctl static s3cp`](/docs/api-reference/iglu/igluctl/#static-s3cp) uploads schemas and generates the schema-list objects loaders rely on in one step:

```bash
igluctl static s3cp /path/to/schemas my-iglu-bucket --region eu-west-1
```

For any other host, generate the schema-list objects locally with `igluctl`, then upload the resulting `schemas` directory to your web root. See the [`igluctl` reference](/docs/api-reference/iglu/igluctl/) for details.

Make sure schema files are accessible to whatever clients you intend to use — typically that means making them publicly readable, since static repos don't support authentication.

## Configure the Iglu resolver

Once your registry is live, add it to your [Iglu resolver configuration](/docs/api-reference/iglu/iglu-resolver/):

```json
{

  "name": "Acme static repo",

  "priority": 0,

  "vendorPrefixes": [ "com.acme" ],

  "connection": {

    "http": {

      "uri": "https://schemas.acme.com"

    }

  }

}
```

Static repos don't require an `apikey`.

---

# Iglu Resolver configuration for Snowplow applications
> Configure Iglu Resolver for schema fetching and validation in Snowplow enrichers and loaders with cache and repository settings.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-resolver/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

Iglu Resolver is a component embedded into many Snowplow applications, including Enrich and loaders. It's responsible for fetching schemas from Iglu registries and validating data against these schemas.

## Configuration

Most of the time, configuring Iglu Resolver means providing a JSON file like this:

```json
{

  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-3",

  "data": {

    "cacheSize": 500,

    "cacheTtl": 600,

    "repositories": [

      {

        "name": "Iglu Central",

        "priority": 0,

        "vendorPrefixes": [ "com.snowplowanalytics" ],

        "connection": {

          "http": {

            "uri": "https://iglucentral.com"

          }

        }

      },

      {

        "name": "Custom Iglu Server",

        "priority": 0,

        "vendorPrefixes": [ "com.snowplowanalytics" ],

        "connection": {

          "http": {

            "uri": "https://${iglu_server_hostname}/api",

            "apikey": "${iglu_server_apikey}"

          }

        }

      }

    ]

  }

}
```

The above configuration assumes Snowplow-authored schemas ([Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central)) will be used in a pipeline, and that you have your own [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) registry hosted at `https://${iglu_server_hostname}/` with a read-rights API key `${iglu_server_apikey}`.

### Configuration parameters

- `cacheSize` determines how many individual schemas the resolver keeps cached.
- `cacheTtl` determines how long a schema can live in the cache before being reloaded, in seconds.
- `repositories` is a JSON array of registries to look up schemas in.
- `priority` and `vendorPrefixes` help the resolver decide which registry to check first for a given schema. See [Registry priority](#registry-priority).

## How schemas are resolved

When the resolver is asked for a schema, it checks its cache first. On a miss, it queries the configured registries in priority order until a registry returns the schema.

### Caching

The resolver caches both successful and failed lookups, keyed per-registry:

- A schema fetched successfully is cached until it's evicted by the LRU algorithm (when the cache reaches `cacheSize` or after `cacheTtl` seconds).
- If a registry responds with "not found", that result is also cached, so the registry won't be queried again for that schema until the entry is evicted.
- If a registry responds with another error (timeout, network error, server fault), the resolver retries that registry up to three more times before marking the schema as missing for that registry.

If `cacheTtl` is set, cache entries (both successful fetches and "not found" results) are re-resolved after the TTL expires. This lets you patch schemas without restarting the pipeline (though patching production schemas isn't recommended). For real-time pipelines, `cacheTtl` also prevents stale "not found" results from persisting for too long — for example, if a schema was missing when first looked up but has since been uploaded.

### Registry priority

For each schema lookup, registries are sorted by:

1. `vendorPrefixes` — the resolver checks registries with a matching `vendorPrefix` first. Other registries aren't skipped, just queried later.
2. Class priority — a hardcoded value per registry type. Embedded registries (which read schemas from the local filesystem or from resources bundled with the application, rather than over HTTP) are always checked before HTTP registries within the same `vendorPrefix` match.
3. `priority` — the user-defined value in your config. Only affects ordering within the same class priority.

Lower numbers mean higher priority. `[0, 1, 2, 3]` is checked left to right.

---

# Igluctl CLI for schema management
> Command-line tool for validating, publishing, and managing JSON schemas in Iglu registries with DDL generation and verification.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/igluctl/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

Iglu provides a CLI application called Igluctl that allows you to perform most common tasks in Iglu Registry.

The available Igluctl commands are:

- `lint` - validate set of JSON schemas for syntax and consistency of their properties

- `static` - work with static Iglu registry

  - `generate` - verify that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for Redshift and Postgres warehouses. Generate DDLs and migrations from set of JSON schemas. If the schema is not evolved correctly and backward incompatible data is sent within transformer's aggregation window, loading would fail for all events.
  - `push` - push set of JSON schemas from static registry to full-featured (Scala Registry for example) one
  - `pull` - pull set of JSON schemas from registry to local folder
  - `deploy` - run entire schema workflow using a config file. This could be used to chain multiple commands, i.e. `lint` followed by `push` and `s3cp`.
  - `s3cp` - copy JSONPaths or schemas to S3 bucket

- `server` - work with an Iglu Server
  - `keygen` - generate read and write API keys on Iglu Server

- `table-check` - will check a given Redshift or Postgres tables against Iglu Server.

- `verify` (since 0.13.0) - work with schemas to check their evolution

  - `redshift` - verify that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for loading into Redshift. It reports the major schema versions within which schema evolution rules were broken.
  - `parquet` - verify that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for parquet transformation (for loading into Databricks). It reports the breaking schema versions.

## Downloading and running Igluctl

Download Igluctl from GitHub releases and unzip the file:

```bash
$ wget https://github.com/snowplow/igluctl/releases/download/0.13.0/igluctl_0.13.0.zip

$ unzip igluctl_0.13.0.zip
```

To run Igluctl you can, for example, can pass the `--help` option to see information on the different commands and flags like this:

```bash
$ ./igluctl --help
```

> **Note:** If you are on Windows, then you'll need to run Igluctl like this:
>
> ```bash
> $ java -jar igluctl --help
> ```
>
> Below and everywhere in documentation you'll find example commands without this `java -jar` prefix, so please remember to add it when running Igluctl.

Note that Igluctl expects [JRE 8](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) or later, and [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or later to run.

## `lint`

`igluctl lint` validates JSON schemas.

It is designed to be run against file-based schema registries with the standard Iglu folder structure:

```text
schemas

└── com.example

    └── my-schema

        └── jsonschema

            ├── 1-0-0

            └── 1-0-1
```

You can validate _all_ the schemas in the registry:

```bash
$ /path/to/igluctl lint /path/to/schema/registry/schemas
```

Alternatively you can validate an individual schema e.g.:

```bash
$ /path/to/igluctl lint /path/to/schema/registry/schemas/com.example_company/example_event/jsonschema/1-0-0
```

Examples of errors that are identified:

- JSON schema has inconsistent self-describing information and path on filesystem
- JSON schema has invalid `$schema` keyword. It should be always set to [Iglu-specific](http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#), while users tend to set it to Draft v4 or even to self-referencing Iglu URI
- JSON schema is invalid against its standard (empty `required`, string `maximum` and similar)
- JSON schema contains properties which contradict each other, like `{"type": "integer", "maxLength": 0}` or `{"maximum": 0, "minimum": 10'}`. These schemas are inherently useless as for some validators there is no JSON instance they can validate

The above cases can very hard to spot without a specialized tool as they are still valid JSON and in last case even valid JSON schemas - so will validate against a standard JSON schema validator.

`lint` has two options:

- `--skip-checks` which will lint without specified linters, given comma separated. To see available linters and their explanations, `$ /path/to/igluctl --help`
- `--skip-schemas` which will lint all the schemas except the schemas passed to this option as a comma separated list. For example running: `/path/to/igluctl lint /path/to/schema/registry/schemas --skip-schemas iglu:com.acme/click/jsonschema/1-0-1,iglu:com.acme/scroll/jsonschema/1-0-1` will lint all schemas in `/path/to/schema/registry/schemas` except the two schemas passed via `--skip-schemas`.

Note: `--severityLevel` option is deprecated and removed as of version 0.4.0.

Below are two groups of linters; allowed to be skipped and not allowed to be skipped. By default, all of them are enabled but Igluctl users can skip any combination of `rootObject`, `unknownFormats`, `numericMinMax`, `stringLength`, `optionalNull`, `description` through `--skip-checks`.

Igluctl let you skip below checks:

| NAME             | DEFINITION                                                                                         |
| ---------------- | -------------------------------------------------------------------------------------------------- |
| `rootObject`     | Check that root of schema has object type and contains properties                                  |
| `unknownFormats` | Check that schema doesn't contain unknown formats                                                  |
| `numericMinMax`  | Check that schema with numeric type contains both minimum and maximum properties                   |
| `stringLength`   | Check that schema with string type contains maxLength property or other ways to extract max length |
| `optionalNull`   | Check that non-required fields have null type                                                      |
| `description`    | Check that property contains description                                                           |

A sample usage could be as following:

```bash
$ ./igluctl lint --skip-checks description,rootObject /path/to/schema/registry/schemas
```

Note that linter names are case sensitive

Igluctl also includes many checks proving that schemas doesn't have conflicting expectations (such as `minimum` value bigger than `maximum`). Schemas with such expectations are valid according to specification, but do not make any sense in real-world use cases. These checks are mandatory and cannot be disabled.

`igluctl lint` will exit with status code 1 if encounter at least one error.

## `static generate`

`igluctl static generate` generates corresponding [Redshift](http://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html) DDL files (`CREATE TABLE` statements) and migration scripts (`ALTER TABLE` statements).

As of version 0.11.0 this command will also validate the compatibility of schema family and display warnings if there is an incompatible evolution.

```bash
$ ./igluctl static generate $INPUT
```

You also can specify directory for output (current dir is used as default):

```bash
$ ./igluctl static generate --output $DDL_DIR $INPUT
```

### Generating migration Redshift table scripts to accommodate updated schema versions

If an input directory is specified with several self-describing JSON schemas with a single REVISION, Igluctl will generate migration scripts to update (`ALTER`) Redshift tables for older schema versions to support the new schema version.

For example, having the following Self-describing JSON schemas as an input:

- schemas/com.acme/click\_event/1-0-0
- schemas/com.acme/click\_event/1-0-1
- schemas/com.acme/click\_event/1-0-2

Igluctl will generate the following migration scripts:

- sql/com.acme/click\_event/1-0-0/1-0-1 to alter table from 1-0-0 to 1-0-1
- sql/com.acme/click\_event/1-0-0/1-0-2 to alter table from 1-0-0 to 1-0-2
- sql/com.acme/click\_event/1-0-1/1-0-2 to alter table from 1-0-1 to 1-0-2

This migrations (and all subsequent table definitions) are aware of column order and will ensure that new columns are added at the end of the table definition. This means that the tables can be updated in-place with single `ALTER TABLE` statements.

### Handling union types

One of the more problematic scenarios to handle when generating Redshift table definitions is handling `UNION` field types e.g. `["integer", "string"]`. Union types will be transformed as most general. In the above example (union of an integer and string type) the corresponding Redshift column will be a `VARCHAR(4096)`.

### Missing schema versions

`static generate` command will check versions of schemas inside `input` as following:

- If user specified folder and one of schemas has no 1-0-0 or misses any other schemas in between (like it has 1-0-0 and 1-0-2) - refuse to do anything (but proceed with –force option)
- If user specified full path to file with schema and this file is not 1-0-0 - just print a warning
- If user specified full path to file with schema and it is 1-0-0 - all good

## `static push`

`igluctl static push` publishes schemas stored locally to a remote [Iglu Server](https://github.com/snowplow/iglu-server).

It accepts three required arguments:

- `host` - Iglu Server host name or IP address with optional port and endpoint. It should conform to the pattern `host:port/path` (or just `host`) **without** http\:// prefix.
- `apikey` - master API key, used to create temporary write and read keys
- `path` - path to your static registry (local folder containing schemas)

Also it accepts optional `--public` argument which makes schemas available without `apikey` header.

```bash
$ ./igluctl static push /path/to/static/registry iglu.acme.com:80/iglu-server f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

## `static pull`

`igluctl static pull` downloads schemas stored on a remote [Iglu Server](https://github.com/snowplow/iglu-server) to a local folder.

It accepts three required arguments:

- `host` - Iglu Server host name or IP address with optional port and endpoint. It should conform to the pattern `host:port/path` (or just `host`) **without** http\:// prefix.
- `apikey` - master API key, used to create temporary write and read keys
- `path` - path to your static registry (local folder to download to)

```bash
$ ./igluctl static pull /path/to/static/registry iglu.acme.com:80/iglu-server f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

## `static s3cp`

`igluctl static s3cp` enables you to upload JSON schemas to chosen S3 bucket. This is helpful for generating a remote Iglu registry which can be served from S3 over http(s).

`igluctl static s3cp` accepts two required arguments and several options:

- `input` - path to your files. Required.
- `bucket` - S3 bucket name. Required.
- `s3path` - optional S3 path to prepend your input root. Usually you don't need it.
- `accessKeyId` - your AWS Access Key Id. This may or may not be required, depending on your preferred authentication option.
- `secretAccessKey` - your AWS Secret Access Key. This may or may not be required, depending on your preferred authentication option.
- `profile` - your AWS profile name. This may or may not be required, depending on your preferred authentication option.
- `region` - AWS S3 region. Default: `us-west-2`
- `skip-schema-lists` - Do not generate and upload schema list objects. If using a static registry for all Snowplow applications, don't enable this setting as some components still require lists to function correctly.

`igluctl static s3cp` tries to closely follow AWS CLI authentication process. First it checks if profile name or `accessKeyId`/`secretAccessKey` pair provided and uses it. If neither of above provided - it looks into `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` environment variables. If above aren't available as well - it `~/.aws/config` file. If all above failed - it exits with error.

## `static deploy`

`igluctl static deploy` performs whole schema workflow at once.

It accepts one required arguments:

- `config` - Path to configuration file

```bash
$ ./igluctl static deploy /path/to/config/file
```

Your configuration file should be a hocon file, following the [reference example](https://github.com/snowplow/igluctl/blob/0.8.0/config/deploy.reference.hocon). For backwards compatibility with previous versions, you could also provide a [self-describing json](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.iglu/igluctl_config/jsonschema/1-0-0).

Example:

```json
  {

    "lint": {

      "skipWarnings": true

      "includedChecks": [

        "rootObject"

        "unknownFormats"

        "numericMinMax"

        "stringLength"

        "optionalNull"

        "description"

        "stringMaxLengthRange"

      ]

    }



    "generate": {

      "dbschema": "atomic"

      "force": false

    }





    "actions": [

      {

        "action": "push"

        "isPublic": true

        "apikey": "bd96b5ff-7eb7-4085-83e0-97ac4954b891"

        "apikey": ${APIKEY_1}

      }

      {

        "action": "s3cp"

        "uploadFormat": "jsonschema"

        "profile": "profile-1"

        "region": "eu-east-2"

      }

    ]



}
```

## `server keygen`

`igluctl server keygen` generates read and write API keys on Iglu Server.

It accepts two required arguments:

- `host` - Iglu Server host name or IP address with optional port and endpoint. It should conform to the pattern `host:port/path` (or just `host`) **without** http\:// prefix.
- `apikey` - master API key, used to create temporary write and read keys

Also it accepts `--vendor-prefix` argument which will be associated with generated key.

```bash
$ ./igluctl server keygen --vendor-prefix com.acme iglu.acme.com:80/iglu-server f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

## `table-check`

`igluctl table-check` will check given RedShift or Postgres schema against an Iglu repository. As of version 0.11.0 it would cross verify the column types as well as names.

It supports two interfaces:

- `igluctl table-check --server <uri>` to check all tables
- `igluctl table-check --resolver <path> --schema <schemaKey>` to check particular table

It also accepts a number of arguments:

```bash
--resolver <path>

    Iglu resolver config path

--schema <schemaKey>

    Schema to check against. It should have iglu:<URI> format

--server <uri>

    Iglu Server URL

--apikey <uuid>

    Iglu Server Read ApiKey (non master)

--dbschema <string>

    Database schema

--host <string>

    Database host address

--port <integer>

    Database port

--dbname <string>

    Database name

--username <string>

    Database username

--password <string>

    Database password
```

```bash
$ ./igluctl table-check --resolver <path> --schema <schemaKey> ...connection parameters
```

or

```bash
$ ./igluctl table-check --server <uri> ...connection params
```

## `verify parquet`

`igluctl verify parquet` verifies that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for parquet transformation (for loading into Databricks). It reports the breaking schema versions.

It accepts one required arguments:

- `input` - path to your schema files.

Example command:

```bash
$ ./igluctl verify parquet /path/to/static/registry
```

Example output:

```text
Breaking change introduced by 'com.acme/product/jsonschema/1-0-2'. Changes: Incompatible type change Long to Double at /item/price

Breaking change introduced by 'com.acme/user/jsonschema/1-0-1'. Changes: Incompatible type change Long to Integer at /id

Breaking change introduced by 'com.acme/item/jsonschema/1-1-0'. Changes: Incompatible type change String to Json at /metadata
```

## `verify redshift`

`igluctl verify redshift` verifies that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for loading into Redshift. It reports the major schema versions within which schema evolution rules were broken.

It accepts two required and one optional arguments:

- `server` - Iglu Server URL.
- `apikey` - Iglu Server Read ApiKey
- `--verbose/-v` - emit detailed report or not (disabled by default)

Example command:

```bash
$ ./igluctl verify redshift --server iglu.acme.com --apikey f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

Example output:

```text
iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-*-*

iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-*-*

iglu:com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-*-*
```

Example command with verbose output:

```bash
$ ./igluctl verify redshift --server iglu.acme.com --apikey f81d4fae-7dec-11d0-a765-00a0c91e6bf6 --verbose
```

Example output:

```text
iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-*-*:

  iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-2: [Incompatible types in column cache_size old RedshiftBigInt new RedshiftDouble]

  iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-3: [Incompatible types in column cache_size old RedshiftBigInt new RedshiftDouble]

iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-*-*:

  iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-0-1: [Incompatible types in column parameters.hash_algorithm old RedshiftChar(3) new RedshiftVarchar(6)]

iglu:com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-*-*:

  iglu:com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-0-1: [Making required column nullable error]
```

---

# Introduction to the Iglu schema registry
> Iglu is the schema registry that powers schema validation in Snowplow pipelines, with public schemas in Iglu Central and private schemas in Iglu Server.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

Iglu is the schema registry that powers schema validation in your Snowplow pipeline. It stores all the [schemas](/docs/fundamentals/schemas/) associated with your events and entities, and serves them to pipeline components for validation.

## How Iglu fits together

- [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) is a private schema registry you run yourself. It exposes a RESTful API for publishing, validating, and serving your custom schemas.
- [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a public registry hosted by Snowplow at [iglucentral.com](https://iglucentral.com/). It contains the schemas for Snowplow's out-of-the-box events and entities.
- [Static repositories](/docs/api-reference/iglu/iglu-repositories/static-repo/) are an alternative to Iglu Server for read-only schema hosting from a static website (for example, on S3).
- The [Iglu Resolver](/docs/api-reference/iglu/iglu-resolver/) is embedded into Snowplow pipeline components (enrichers, loaders) and fetches schemas from one or more configured registries.
- [Igluctl](/docs/api-reference/iglu/igluctl/) is a command-line tool for validating, publishing, and managing schemas.

---

# Manage schemas using Iglu Server
> Manage schemas with Iglu Server or host a static Iglu registry in Amazon S3 or Google Cloud Storage for self-hosted Snowplow deployments.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/manage-schemas/

> **Info:** If you're a Snowplow CDI customer, you don't need to manage Iglu directly — it's included in your pipeline. Use [Event Studio](/docs/event-studio/data-structures/) or the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to manage your schemas. This page is for Self-Hosted customers.

To manage your [schemas](/docs/fundamentals/schemas/), you need an [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) (you'll already have one if you followed the [Snowplow Self-Hosted quick start](/docs/get-started/self-hosted/)). Alternatively, you can host a [static Iglu registry](/docs/api-reference/iglu/iglu-repositories/static-repo/).

## Create a schema

First, design the schema for your custom event or entity. For example:

```json
{

     "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

     "description": "Schema for a button click event",

     "self": {

         "vendor": "com.snowplowanalytics",

         "name": "button_click",

         "format": "jsonschema",

         "version": "1-0-0"

     },

     "type": "object",

     "properties": {

         "id": {

             "type": "string",

             "minLength": 1

         },

         "target": {

             "type": "string"

         },

         "content": {

             "type": "string"

         }

     },

     "required": ["id"],

     "additionalProperties": false

 }
```

Next, save this schema in the following folder structure, with a filename of `1-0-0` (without any extension):

```text
schemas

└── com.snowplowanalytics

    └── button_click

        └── jsonschema

            └── 1-0-0
```

> **Tip:** If you update the `vendor` or the `name` in the example, you should update the above path too.

Finally, to upload your schema to your Iglu registry, use [Igluctl](/docs/api-reference/iglu/igluctl/):

```bash
igluctl static push --public <local path to schemas> <Iglu server endpoint> <iglu_super_api_key>
```

See the [Igluctl reference page](/docs/api-reference/iglu/igluctl/#static-push) for more information on the `static push` command.

## Versioning schemas

When evolving your [schema](/docs/fundamentals/schemas/) and [uploading](/docs/api-reference/iglu/manage-schemas/) it to your [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/), you will need to choose how to increment its version.

There are two kinds of schema changes:

- **Non-breaking** — a non-breaking change is backward compatible with historical data and increments the `patch` version (for example, `1-0-0` to `1-0-1`) or the `minor` version (for example, `1-0-0` to `1-1-0`).
- **Breaking** — a breaking change is not backward compatible with historical data and increments the `major` version (for example, `1-0-0` to `2-0-0`).

Different data warehouses handle schema evolution slightly differently. Use the table below as a guide for incrementing the schema version appropriately.

|                                              | Redshift     | Snowflake, BigQuery, Databricks |
| -------------------------------------------- | ------------ | ------------------------------- |
| **Add / remove / rename an optional field**  | Non-breaking | Non-breaking                    |
| **Add / remove / rename a required field**   | Breaking     | Breaking                        |
| **Change a field from optional to required** | Breaking     | Breaking                        |
| **Change a field from required to optional** | Breaking     | Non-breaking                    |
| **Change the type of an existing field**     | Breaking     | Breaking                        |
| **Change the size of an existing field**     | Non-breaking | Non-breaking                    |

> **Note:** In Redshift and Databricks, changing _size_ may also mean _type_ change. For example, changing the `maximum` integer from `30000` to `100000`. See our documentation on [how schemas translate to database types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

---

# JSON Schema reference
> Complete reference for JSON Schema draft 4 features supported in Snowplow data structures, including validation keywords and type definitions.
> Source: https://docs.snowplow.io/docs/api-reference/json-schema-reference/

[Snowplow schemas](/docs/fundamentals/schemas/) are based on the [JSON Schema](https://json-schema.org/) standard ([draft 4](https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00)). This reference provides comprehensive documentation for all JSON Schema features that are supported in Snowplow.

Understanding the full capabilities of JSON Schema allows you to create more precise and robust [data structures](/docs/fundamentals/schemas/) that ensure your data quality and provide clear documentation for your tracking implementation.

## Schema structure

Every Snowplow schema must follow this basic structure with Snowplow-specific metadata and JSON Schema validation rules:

```json
{

  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

  "description": "Human-readable description of the schema purpose",

  "self": {

    "vendor": "com.example",

    "name": "schema_name",

    "format": "jsonschema",

    "version": "1-0-0"

  },

  "type": "object",

  "properties": {

    // Field definitions go here

  },

  "additionalProperties": false,

  "required": ["required_field_name"]

}
```

### Required components

Every Snowplow schema must include these components:

- **`$schema`**: Must be `"http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#"`

- **`self`** object containing:

  - **`vendor`**: Your organization identifier (e.g., `"com.example"`)
  - **`name`**: The schema name
  - **`format`**: Must be `"jsonschema"`
  - **`version`**: Semantic version (e.g., `"1-0-0"`)

- **`type`**: Must be `"object"` for the root level

### Optional components

These components are optional but commonly used:

- **`description`**: Human-readable description of the schema purpose (highly recommended)
- **`properties`**: Object defining the fields and their validation rules
- **`additionalProperties`**: Whether additional properties are allowed (commonly set to `false`)
- **`required`**: Array of required field names
- **`minProperties`** / **`maxProperties`**: Constraints on number of properties

## Core validation keywords

### Type validation

The `type` keyword specifies the expected data type for a value. Snowplow supports all JSON Schema primitive types:

#### String type

```json
{

  "user_name": {

    "type": "string",

    "description": "The user's display name"

  }

}
```

#### Number and integer types

```json
{

  "price": {

    "type": "number",

    "description": "Product price in USD"

  },

  "quantity": {

    "type": "integer",

    "description": "Number of items purchased"

  }

}
```

#### Boolean type

```json
{

  "is_premium": {

    "type": "boolean",

    "description": "Whether the user has a premium account"

  }

}
```

#### Array type

```json
{

  "tags": {

    "type": "array",

    "description": "Product tags",

    "items": {

      "type": "string"

    }

  }

}
```

#### Object type

```json
{

  "address": {

    "type": "object",

    "description": "User's shipping address",

    "properties": {

      "street": {"type": "string"},

      "city": {"type": "string"},

      "postal_code": {"type": "string"}

    }

  }

}
```

#### Null type

```json
{

  "middle_name": {

    "type": ["string", "null"],

    "description": "User's middle name (optional)"

  }

}
```

### Multiple types

You can specify multiple acceptable types using an array:

```json
{

  "user_id": {

    "type": ["string", "integer"],

    "description": "User identifier (string or numeric)"

  },

  "optional_field": {

    "type": ["string", "null"],

    "description": "Optional text field"

  }

}
```

## String validation

### Length constraints

Control the minimum and maximum length of string values:

```json
{

  "username": {

    "type": "string",

    "minLength": 3,

    "maxLength": 20,

    "description": "Username between 3-20 characters"

  },

  "password": {

    "type": "string",

    "minLength": 8,

    "description": "Password must be at least 8 characters"

  }

}
```

### Enumeration

Restrict values to a specific set of allowed strings:

```json
{

  "status": {

    "type": "string",

    "enum": ["active", "inactive", "pending", "suspended"],

    "description": "Account status"

  },

  "color": {

    "type": "string",

    "enum": ["red", "green", "blue", "yellow"],

    "description": "Primary color selection"

  }

}
```

### Pattern matching

Use regular expressions to validate string format:

```json
{

  "product_code": {

    "type": "string",

    "pattern": "^[A-Z]{2}-\\d{4}$",

    "description": "Product code format (e.g., AB-1234)"

  },

  "phone_number": {

    "type": "string",

    "pattern": "^\\+?[1-9]\\d{1,14}$",

    "description": "International phone number format"

  }

}
```

> **Tip:** For common formats like email addresses, URLs, and dates, prefer using the `format` keyword instead of regular expressions for better readability and standardized validation.

### Format validation

Use the `format` keyword to validate common string formats:

```json
{

  "email": {

    "type": "string",

    "format": "email",

    "description": "Valid email address"

  },

  "website": {

    "type": "string",

    "format": "uri",

    "description": "Website URL"

  },

  "server_ip": {

    "type": "string",

    "format": "ipv4",

    "description": "IPv4 address of the server"

  },

  "created_at": {

    "type": "string",

    "format": "date-time",

    "description": "ISO 8601 timestamp"

  },

  "user_id": {

    "type": "string",

    "format": "uuid",

    "description": "UUID identifier"

  }

}
```

#### Supported format values

- **`uri`**: Uniform Resource Identifier
- **`ipv4`**: IPv4 address (e.g., "192.168.1.1")
- **`ipv6`**: IPv6 address
- **`email`**: Email address
- **`date-time`**: ISO 8601 date-time (e.g., "2023-12-25T10:30:00Z")
- **`date`**: ISO 8601 date (e.g., "2023-12-25")
- **`hostname`**: Internet hostname
- **`uuid`**: UUID string

## Numeric validation

### Range constraints

Set minimum and maximum values for numbers and integers:

```json
{

  "age": {

    "type": "integer",

    "minimum": 0,

    "maximum": 150,

    "description": "Person's age in years"

  },

  "discount_rate": {

    "type": "number",

    "minimum": 0,

    "maximum": 1,

    "description": "Discount rate between 0 and 1"

  }

}
```

### Multiple constraints

Combine multiple numeric validations:

```json
{

  "rating": {

    "type": "number",

    "minimum": 1,

    "maximum": 5,

    "multipleOf": 0.5,

    "description": "Star rating in half-point increments"

  }

}
```

## Array validation

### Length constraints

Control the size of arrays:

```json
{

  "favorite_colors": {

    "type": "array",

    "minItems": 1,

    "maxItems": 5,

    "description": "User's favorite colors (1-5 selections)",

    "items": {

      "type": "string",

      "enum": ["red", "blue", "green", "yellow", "purple", "orange"]

    }

  }

}
```

### Item validation

Define validation rules for array items:

```json
{

  "purchase_items": {

    "type": "array",

    "description": "Items in the purchase",

    "items": {

      "type": "object",

      "properties": {

        "product_id": {"type": "string"},

        "quantity": {"type": "integer", "minimum": 1},

        "price": {"type": "number", "minimum": 0}

      },

      "required": ["product_id", "quantity", "price"],

      "additionalProperties": false

    }

  }

}
```

### Unique items

Ensure all array items are unique:

```json
{

  "user_tags": {

    "type": "array",

    "uniqueItems": true,

    "description": "Unique tags assigned to user",

    "items": {

      "type": "string"

    }

  }

}
```

## Object validation

### Property requirements

Specify which object properties are required:

```json
{

  "user_profile": {

    "type": "object",

    "properties": {

      "first_name": {"type": "string"},

      "last_name": {"type": "string"},

      "email": {"type": "string"},

      "phone": {"type": ["string", "null"]}

    },

    "required": ["first_name", "last_name", "email"],

    "additionalProperties": false

  }

}
```

### Additional properties

Control whether additional properties are allowed:

```json
{

  "strict_object": {

    "type": "object",

    "properties": {

      "name": {"type": "string"},

      "value": {"type": "number"}

    },

    "additionalProperties": false,

    "description": "Only name and value properties allowed"

  },

  "flexible_object": {

    "type": "object",

    "properties": {

      "core_field": {"type": "string"}

    },

    "additionalProperties": true,

    "description": "Additional properties are permitted"

  }

}
```

### Property count constraints

Limit the number of properties in an object:

```json
{

  "metadata": {

    "type": "object",

    "minProperties": 1,

    "maxProperties": 10,

    "additionalProperties": {"type": "string"},

    "description": "Metadata with 1-10 string properties"

  }

}
```

## Advanced validation patterns

### Schema composition

Use `oneOf` and `anyOf` to create flexible validation rules:

#### Using oneOf

Validate that data matches exactly one of several schemas:

```json
{

  "contact_info": {

    "type": "object",

    "oneOf": [

      {

        "properties": {

          "type": {"enum": ["email"]},

          "email": {"type": "string", "format": "email"}

        },

        "required": ["type", "email"],

        "additionalProperties": false

      },

      {

        "properties": {

          "type": {"enum": ["phone"]},

          "phone": {"type": "string", "pattern": "^\\+?[1-9]\\d{1,14}$"}

        },

        "required": ["type", "phone"],

        "additionalProperties": false

      },

      {

        "properties": {

          "type": {"enum": ["address"]},

          "street": {"type": "string"},

          "city": {"type": "string"},

          "postal_code": {"type": "string"}

        },

        "required": ["type", "street", "city", "postal_code"],

        "additionalProperties": false

      }

    ]

  }

}
```

#### Using anyOf

Validate that data matches one or more of several schemas:

```json
{

  "user_permissions": {

    "type": "object",

    "anyOf": [

      {

        "properties": {

          "can_read": {"type": "boolean"}

        },

        "required": ["can_read"]

      },

      {

        "properties": {

          "can_write": {"type": "boolean"}

        },

        "required": ["can_write"]

      },

      {

        "properties": {

          "can_admin": {"type": "boolean"}

        },

        "required": ["can_admin"]

      }

    ]

  }

}
```

## Limitations and unsupported features

While Snowplow supports most JSON Schema Draft 4 features, there are some limitations to be aware of:

- **`$ref`**: Schema references are not supported in property definitions
- **`allOf`**: Schema intersection is not supported
- **`not`**: Negation validation is not supported
- **`dependencies`**: Property dependencies are not supported
- **`exclusiveMinimum`** and **`exclusiveMaximum`**: Exclusive bounds are not supported

Instead of unsupported features, use these approaches:

```json
{

  // Instead of $ref, define inline schemas

  "address": {

    "type": "object",

    "properties": {

      "street": {"type": "string"},

      "city": {"type": "string"},

      "country": {"type": "string", "enum": ["US", "CA", "UK", "DE"]}

    },

    "required": ["street", "city", "country"],

    "additionalProperties": false

  },



  // Instead of exclusiveMinimum/exclusiveMaximum, use minimum/maximum with adjusted values

  "percentage": {

    "type": "number",

    "minimum": 0,

    "maximum": 99.99,

    "description": "Percentage value (0 to less than 100)"

  },



  // Use format validation for common patterns

  "created_date": {

    "type": "string",

    "format": "date-time",

    "description": "ISO 8601 timestamp"

  }

}
```

---

# BigQuery Loader configuration reference
> Configure BigQuery Streaming Loader with BigQuery, Kinesis, and Pub/Sub settings for streaming enriched Snowplow events.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/

The configuration reference in this page is written for BigQuery Loader `2.2.1`

### License

The BigQuery Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {

  "accept": true

}
```

### BigQuery configuration

| Parameter                 | Description                                                                                                                                                                                               |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.project`     | Required. The GCP project to which the BigQuery dataset belongs                                                                                                                                           |
| `output.good.dataset`     | Required. The BigQuery dataset to which events will be loaded                                                                                                                                             |
| `output.good.table`       | Optional. Default value `events`. Name to use for the events table                                                                                                                                        |
| `output.good.credentials` | Optional. Service account credentials (JSON). If not set, default credentials will be sourced from the usual locations, e.g. file pointed to by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable |

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-bigquery-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                       |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |
| `output.bad.maxRetries` (since 2.1.0)               | Optional. Default value 10. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                     |

**GCP:**

| Parameter                                               | Description                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `input.subscription`                                    | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                           |
| `input.durationPerAckExtension`                         | Optional. Default value `15 seconds`. Pub/Sub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                                  |
| `input.minRemainingAckDeadline`                         | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `15 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `1.5 seconds` left of the remaining deadline, before re-extending the message deadline. |
| `input.retries.transientErrors.delay` (since 2.1.0)     | Optional. Default value `100 millis`. Backoff delay for follow-up attempts after transient errors.                                                                                                                                                                                                                                                                       |
| `input.retries.transientErrors.attempts` (since 2.1.0)  | Optional. Default value `10`. Maximum number of attempts, after which the loader will crash and exit.                                                                                                                                                                                                                                                                    |
| `output.bad.topic`                                      | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                      |
| `output.bad.batchSize`                                  | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                        |
| `output.bad.requestByteThreshold`                       | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                           |
| `output.bad.maxRecordSize`                              | Optional. Default value 9000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                            |
| `output.retries.transientErrors.delay` (since 2.1.0)    | Optional. Default value `100 millis`. Backoff delay for follow-up attempts after transient errors.                                                                                                                                                                                                                                                                       |
| `output.retries.transientErrors.attempts` (since 2.1.0) | Optional. Default value `10`. Maximum number of attempts, after which the loader will crash and exit.                                                                                                                                                                                                                                                                    |

***

## Other configuration options

| Parameter                                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batching.maxBytes`                       | Optional. Default value `10000000`. Events are emitted to BigQuery when the batch reaches this size in bytes                                                                                                                                                                                                                                                                                                                                                          |
| `batching.maxDelay`                       | Optional. Default value `1 second`. Events are emitted to BigQuery after a maximum of this duration, even if the `maxBytes` size has not been reached                                                                                                                                                                                                                                                                                                                 |
| `batching.writeBatchConcurrency`          | Optional. Default value 2. How many batches can we send simultaneously over the network to BigQuery                                                                                                                                                                                                                                                                                                                                                                   |
| `cpuParallelism.parseBytesFactor`         | Optional. Default value `0.1`.Controls how many batches of bytes we can parse into enriched events simultaneously.E.g. If there are 2 cores and `parseBytesFactor = 0.1` then only one batch gets processed at a time.Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                                                                                |
| `cpuParallelism.transformFactor`          | Optional. Default value `0.75`.Controls how many batches of enriched events we can transform into BigQuery format simultaneously.E.g. If there are 4 cores and `transformFactor = 0.75` then 3 batches gets processed in parallel.Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                                                                    |
| `retries.setupErrors.delay`               | Optional. Default value `30 seconds`.Configures exponential backoff on errors related to how BigQuery is set up for this loader.Examples include authentication errors and permissions errors.This class of errors are reported periodically to the monitoring webhook.                                                                                                                                                                                               |
| `retries.transientErrors.delay`           | Optional. Default value `1 second`.Configures exponential backoff on errors that are likely to be transient.Examples include server errors and network errors.                                                                                                                                                                                                                                                                                                        |
| `retries.transientErrors.attempts`        | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                                                                                                                                  |
| `retries.alterTableWait.delay`            | Optional. Default value `1 second`.Configures backoff when waiting for the BigQuery Writer to detect that the loader has altered the table by adding new columns.                                                                                                                                                                                                                                                                                                     |
| `retries.tooManyColumns.delay`            | Optional. Default value `300 seconds`.Relevant when the BigQuery table is close to exceeding the limit on max allowed columns in a single table. The loader will ignore a failure to alter the table due to too many columns, and it will continue to run. Some events will inevitably go to the failed events output topic until new columns have been added. This parameter configures how often the loader will retry to alter the table after an earlier failure. |
| `skipSchemas`                             | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`.A list of schemas that won't be loaded to BigQuery.This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                                                                                                                         |
| `legacyColumnMode`                        | Optional. Default value `false`.When this mode is enabled, the loader uses the legacy column style used by the v1 BigQuery loader.For example, an entity for a `1-0-0` schema is loaded into a column ending in `_1_0_0`, instead of a column ending in `_1`.This feature could be helpful when migrating from the v1 loader to the v2 loader.                                                                                                                        |
| `legacyColumns`                           | Optional, e.g. `["iglu:com.example/legacy/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/legacy/jsonschema/1--"]`.Schemas for which to use the legacy column style used by the v1 BigQuery loader, even when `legacyColumnMode` is disabled.                                                                                                                                                                                                               |
| `exitOnMissingIgluSchema`                 | Optional. Default value `true`.Whether the loader should crash and exit if it fails to resolve an Iglu Schema.We recommend `true` because Snowplow enriched events have already passed validation, so a missing schema normally indicates an error that needs addressing.Change to `false` so events go the failed events stream instead of crashing the loader.                                                                                                      |
| `decompression.maxBytesInBatch`           | Optional. Default value `5242880` (5 MB).The loader automatically detects and decompresses zstd- or gzip-compressed source messages. Uncompressed messages are unaffected.The loader uses this as a cutoff when incrementally adding decompressed events to a batch, and emits the batch as soon as it reaches this size. This protects the loader's memory because a small compressed message can expand into a much larger payload.                                 |
| `decompression.maxBytesSinglePayload`     | Optional. Default value `10000000` (10 MB).For zstd- or gzip-compressed source messages, this is the maximum size of a single payload after decompression. The loader emits a [size violation failed event](/docs/api-reference/failed-events/#size-violation) for any payload that exceeds this size.                                                                                                                                                                |
| `monitoring.metrics.statsd.hostname`      | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.metrics.statsd.port`          | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                                                                                                                          |
| `monitoring.metrics.statsd.tags.*`        | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.metrics.statsd.period`        | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.prefix`        | Optional. Default `snowplow.bigquery-loader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.prometheus.tags.*`    | Optional. A map of key/value pairs added as common labels on every Prometheus metric. The loader exposes these metrics at the `/metrics` endpoint on the health probe port.                                                                                                                                                                                                                                                                                           |
| `monitoring.webhook.endpoint`             | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how BigQuery is set up for this loader.                                                                                                                                                                                                                                                                                        |
| `monitoring.webhook.tags.*`               | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.webhook.heartbeat.*`          | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.healthProbe.port`             | Optional. Default value `8000`. Open a HTTP server that returns OK only if the app is healthy.                                                                                                                                                                                                                                                                                                                                                                        |
| `monitoring.healthProbe.unhealthyLatency` | Optional. Default value `5 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time.                                                                                                                                                                                                                                                                                                                       |
| `monitoring.sentry.dsn`                   | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.sentry.tags.*`                | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                                                    |
| `telemetry.disable`                       | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                                                             |
| `telemetry.userProvidedId`                | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                                                   |
| `http.client.maxConnectionsPerServer`     | Optional. Default value 4. Configures the internal HTTP client used for iglu resolver, alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time.                                                                                                                                                                                                                                                                           |

---

# BigQuery Streaming Loader
> Stream Snowplow events to BigQuery from Kinesis or Pub/Sub with real-time loading and schema evolution.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/

The BigQuery Streaming Loader is an application that loads Snowplow events to BigQuery.

**AWS:**

On AWS, the BigQuery Streaming Loader continually pulls events from Kinesis and writes to BigQuery using the [BigQuery Storage API](https://cloud.google.com/bigquery/docs/write-api).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>BigQuery Loader</b>"}}
subgraph bigquery [BigQuery]
  table[("<b>Events table</b>")]
end
stream-->loader-->|BigQuery Storage API|bigquery
```

The BigQuery Loader is published as a Docker image which you can run on any AWS VM.

```bash
docker pull snowplow/bigquery-loader-kinesis:2.2.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

snowplow/bigquery-loader-kinesis:2.2.1 \

--config=/myconfig/loader.hocon \

--iglu-config /myconfig/iglu.hocon
```

Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

**GCP:**

On GCP, the BigQuery Streaming Loader continually pulls events from Pub/Sub and writes to BigQuery using the [BigQuery Storage API](https://cloud.google.com/bigquery/docs/write-api).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>BigQuery Loader</b>"}}
subgraph bigquery [BigQuery]
  table[("<b>Events table</b>")]
end
stream-->loader-->|BigQuery Storage API|bigquery
```

The BigQuery Loader is published as a Docker image which you can run on any GCP VM.

```bash
docker pull snowplow/bigquery-loader-pubsub:2.2.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

snowplow/bigquery-loader-pubsub:2.2.1 \

--config=/myconfig/loader.hocon \

--iglu-config /myconfig/iglu.hocon
```

Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

***

> **Tip:** For more information on how events are stored in BigQuery, check the [mapping between Snowplow schemas and the corresponding BigQuery column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=bigquery).

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-bigquery-loader/blob/master/config/config.kinesis.minimal.hocon&#xA;>)

**GCP:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-bigquery-loader/blob/master/config/config.pubsub.minimal.hocon&#xA;>)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for all possible configuration parameters.

### Iglu

The BigQuery Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

## Metrics

The BigQuery Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:

| Metric               | Definition                                                                                                                                                     |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_good`        | A count of events that are successfully written to BigQuery.                                                                                                   |
| `events_bad`         | A count of failed events that could not be loaded, and were instead sent to the bad output stream.                                                             |
| `latency_millis`     | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader.                  |
| `e2e_latency_millis` | The end-to-end latency of the snowplow pipeline. The time in milliseconds from when an event was received by the collector, until it is written into BigQuery. |

See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for how to configure the StatsD receiver.

**Telemetry notice**

By default, Snowplow collects telemetry data for BigQuery Loader (since version 2.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# BigQuery Loader 1.0.x upgrade guide
> Upgrade BigQuery Loader from 0.6.x to 1.0.x with HOCON config, new load_tstamp field, and StreamLoader migration steps.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/1-0-x-upgrade-guide/

## Configuration

The only breaking change from the 0.6.x series is the new format of the configuration file. That used to be a self-describing JSON but is now HOCON. Additionally, some app-specific command-line arguments have been incorporated into the config, such as Repeater's `--failedInsertsSub` option. For more details, see the [setup guide](/docs/api-reference/loaders-storage-targets/bigquery-loader/) and [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/configuration-reference/).

Using Repeater as an example, if your configuration for 0.6.x looked like this:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.storage/bigquery_config/jsonschema/1-0-0",

  "data": {

    "name": "Alpha BigQuery test",

    "id": "31b1559d-d319-4023-aaae-97698238d808",

    "projectId": "com-acme",

    "datasetId": "snowplow",

    "tableId": "events",

    "input": "enriched-sub",

    "typesTopic": "types-topic",

    "typesSubscription": "types-sub",

    "badRows": "bad-topic",

    "failedInserts": "failed-inserts-topic",

    "load": {

      "mode": "STREAMING_INSERTS",

      "retry": false

     },

     "purpose": "ENRICHED_EVENTS"

  }

}
```

it will now look like this:

```json
{

  "projectId": "com-acme"



  "loader": {

    "input": {

      "subscription": "enriched-sub"

    }



    "output": {

      "good": {

        "datasetId": "snowplow"

        "tableId": "events"

      }



      "bad": {

        "topic": "bad-topic"

      }



      "types": {

        "topic": "types-topic"

      }



      "failedInserts": {

        "topic": "failed-inserts-topic"

      }

    }

  }



  "mutator": {

    "input": {

      "subscription": "types-sub"

    }



    "output": {

      "good": ${loader.output.good} # will be automatically inferred

    }

  }



  "repeater": {

    "input": {

      "subscription": "failed-inserts-sub"

    }



    "output": {

      "good": ${loader.output.good} # will be automatically inferred



      "deadLetters": {

        "bucket": "gs://dead-letter-bucket"

      }

    }

  }



  "monitoring": {} # disabled

}
```

And instead of running it like this:

```bash
$ ./snowplow-bigquery-repeater \

    --config=$CONFIG \

    --resolver=$RESOLVER \

    --failedInsertsSub="failed-inserts-sub" \

    --deadEndBucket="gs://dead-letter-bucket"

    --desperatesBufferSize=20 \

    --desperatesWindow=20 \

    --backoffPeriod=900 \

    --verbose
```

you will run it like this:

```bash
$ docker run \

  -v /path/to/resolver.json:/resolver.json \

  snowplow/snowplow-bigquery-repeater:1.0.1 \

    --config=$CONFIG \

    --resolver=/resolver.json \

    --bufferSize=20 \

    --timeout=20 \

    --backoffPeriod=900 \

    --verbose
```

## New events table field

The first time you deploy Mutator 1.0.0 it will add a new column to your events table: `load_tstamp`. This represents the exact moment when the row was inserted into BigQuery. It shows you when events have arrived in the warehouse, which makes it possible to use incremental processing of newly arrived data in your downstream data modeling.

Depending on your traffic volume and pattern, there might be a short time period in which the loader app cannot write to BigQuery because the new column hasn't propagated and is not yet visible to all workers. For that reason, **we recommend that you upgrade Mutator first**.

## Migrating to StreamLoader

StreamLoader has been built as a standalone application, replacing Apache Beam and no longer requires you to use Dataflow.

Depending on your data volume and traffic patterns, this might lead to significant cost reductions. However, by migrating away from Dataflow, you no longer benefit from its exactly-once processing guarantees. As such, there could be a slight increase in the number of duplicate events loaded into BigQuery.

Duplicate events generally are to be expected in a Snowplow pipeline, which provides an at-least-once guarantee.

In our tests, we found that duplicates arise only during extreme autoscaling of the loader, eg if your pipeline has a sudden extreme spike in events. Aside from autoscaling events, we found the number of duplicate rows to be very low, however this depends on the type of worker infrastructure you use.

---

# BigQuery Loader 2.0.0 upgrade guide
> Guide for upgrading to BigQuery Loader 2.0.0, including configuration changes, new column naming strategy, and recovery columns for schema evolution.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/2-0-0-upgrade-guide/

## Configuration

BigQuery Loader 2.0.0 brings changes to the loading setup. It is no longer necessary to configure and deploy three independent applications (Loader, Repeater and Mutator in [1.X](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/)) in order to load your data to BigQuery. Starting from 2.0.0, only one application is needed, which naturally introduces some breaking changes to the configuration file structure.

See the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for all possible configuration parameters and the minimal [configuration samples](https://github.com/snowplow-incubator/snowplow-bigquery-loader/blob/v2/config) for each of supported cloud environments.

## Infrastructure

Apart from Repeater and Mutator, other infrastructure components have become obsolete:

- The `types` PubSub topic connecting Loader and Mutator.
- The `failedInserts` PubSub topic connecting Loader and Repeater.
- The `deadLetter` GCS bucket used by Repeater to store data that repeatedly failed to be inserted into BigQuery.

This means that failed events are now written to the failed events PubSub topic, configured as `output.bad.topic`, rather than directly to the GCS bucket as before. This change was made to consolidate all types of event failures into a single place.

## Events table format

Starting from 2.0.0, BigQuery Loader changes its output column naming strategy. For example, for [ad\_click event](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_click_event/jsonschema/1-0-0):

- Before an upgrade, the corresponding column would be named `unstruct_event_com_snowplowanalytics_snowplow_media_ad_click_event_1_0_0`.
- After an upgrade, new column will be named `unstruct_event_com_snowplowanalytics_snowplow_media_ad_click_event_1`.

All self-describing events and entities will be loaded to new "major version"-oriented columns. Old "full version"-oriented columns would remain unchanged, but no new data would be loaded into them (the 2.0.0 loader just ignores these columns).

The new column naming scheme has several advantages:

- Fewer columns created (BigQuery has a limit on the total number of columns)
- No need to update data models (or use complex macros) every time a new minor version of a schema is created

The catch is that you have to follow the rules of schema evolution more strictly to ensure data from different schema versions can fit in the same column — see below.

> **Tip:** If you are using [Snowplow dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/), they will automatically consolidate the data between `_1_0_0` and `_1` style columns, because they look at the major version prefix (e.g. `_1`), which is common to both.
>
> If you are not using Snowplow dbt models but still use dbt, you can employ [this macro](https://github.com/snowplow/dbt-snowplow-utils#combine_column_versions-source) to manually aggregate the data across old and new columns.

### Enable legacy mode for the old table format

To simplify migration to the new table format, it is possible to run the 2.x loader in legacy mode, so it loads self-describing events and entities using the old column names of the 1.x loader.

**Option 1:** In the configuration file, set `legacyColumnMode` to `true`. When this mode is enabled, the loader uses the legacy column style for all self-describing events and entities.

**Option 2:** In the configuration file, set `legacyColumns` to list specific schemas for which to use the legacy column style. This list is used when `legacyColumnMode` is `false` (the default).

For example:

```json
"legacyColumns": [

  "iglu:com.example/legacy_a/jsonschema/1-0-0",

  "iglu:com.example/legacy_b/jsonschema/1-*-*"

]
```

## Recovery columns

### What is schema evolution?

One of Snowplow’s key features is the ability to [define custom schemas and validate events against them](/docs/fundamentals/schemas/). Over time, users often evolve the schemas, e.g. by adding new fields or changing existing fields. To accommodate these changes, BigQuery Loader 2.0.0 automatically adjusts the database tables in the warehouse accordingly.

There are two main types of schema changes:

**Breaking**: The schema version has to be changed in a major way (`1-2-3` → `2-0-0`). As of BigQuery Loader 2.0.0, each major schema version has its own column (`..._1`, `..._2`, etc, for example: `contexts_com.snowplowanalytics_ad_click_1`).

**Non-breaking**: The schema version can be changed in a minor way (`1-2-3` → `1-3-0` or `1-2-3` → `1-2-4`). Data is stored in the same database column.

Loader tries to format the incoming data according to the latest version of the schema it saw (for a given major version, e.g. `1-*-*`). For example, if a batch contains events with schema versions `1-0-0`, `1-0-1` and `1-0-2`, the loader derives the output schema based on version `1-0-2`. Then the loader instructs BigQuery to adjust the database column and load the data.

### Recovering from invalid schema evolution

Let's consider these two schemas as an example of breaking schema evolution (changing the type of a field from `integer` to `string`) using the same major version (`1-0-0` and `1-0-1`):

```json
{

   // 1-0-0

   "properties": {

      "a": {"type": "integer"}

   }

}
```

```json
{

   // 1-0-1

   "properties": {

      "a": {"type": "string"}

   }

}
```

With BigQuery Loader 1.x, data for each version would go to its own column — no issue. With BigQuery Loader 2.x, there is only one column. But strings and integers can’t coexist!

To avoid crashing or losing data, BigQuery Loader 2.0.0 proceeds by creating a new column for the data with schema `1-0-1`, e.g. `contexts_com_snowplowanalytics_ad_click_1_0_1_recovered_9999999`, where:

- `1_0_1` is the version of the offending schema;
- `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one).

If you create a new schema `1-0-2` that reverts the offending changes and is again compatible with `1-0-0`, the data for events with that schema will be written to the original column as expected.

> **Tip:** You might find that some of your schemas were evolved incorrectly in the past, which results in the creation of these “recovery” columns after the upgrade. To address this for a given schema, create a new _minor_ schema version that reverts the breaking changes introduced in previous versions. (Or, if you want to keep the breaking change, create a new _major_ schema version.) You can set it to [supersede](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded) the previous version(s), so that events are automatically validated against the new schema.

> **Note:** If events with incorrectly evolved schemas never arrive, then the recovery column would not be created.

You can read more about schema evolution and how recovery columns work [here](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=bigquery#versioning).

---

# Upgrade guides for the BigQuery Loader
> Upgrade guides for BigQuery Loader with breaking changes, migration steps, and compatibility notes for major versions.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/

This section contains information to help you upgrade to newer versions of the BigQuery Loader.

---

# Databricks Streaming Loader configuration reference
> Configure Databricks Streaming Loader with Unity Catalog, Kinesis, Pub/Sub, and Kafka settings for lakehouse streaming.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/configuration-reference/

The configuration reference in this page is written for Databricks Streaming Loader `0.5.0`

### License

The Databricks Streaming Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {

  "accept": true

}
```

### Databricks configuration

| Parameter                        | Description                                                                                                                                                                                |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `output.good.host`               | Required, e.g. `https://{workspace-id}.cloud.databricks.com`. URL of the Databricks workspace.                                                                                             |
| `output.good.catalog`            | Required. Name of the Databricks catalog containing the volume.                                                                                                                            |
| `output.good.schema`             | Required. Name of the Databricks schema containing the volume.                                                                                                                             |
| `output.good.volume`             | Required. Name of the Databricks volume to which this loader will upload staging files. This must be an [external](https://docs.databricks.com/aws/en/volumes/managed-vs-external) volume. |
| `output.good.token`              | Required if using PAT authentication. A Databricks [personal access token](https://docs.databricks.com/aws/en/dev-tools/auth).                                                             |
| `output.good.oauth.clientId`     | Required if using OAUTH authentication. The client ID for a Databricks [service principal](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m).                                   |
| `output.good.oauth.clientSecret` | Required if using OAUTH authentication. The client secret for a Databricks [service principal](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m).                               |
| `output.good.compression`        | Optional. Default value `snappy`. Compression algorithm for the uploaded staging parquet files.                                                                                            |
| `output.good.httpTimeout`        | Optional. Default value `20 seconds`. Timeout duration of Databricks SDK's HTTP client.                                                                                                    |

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-databricks-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                     |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `input.debounceCheckpoints`                         | Optional. Default value `10 seconds`. How frequently to checkpoint our progress to the DynamoDB table. By increasing this value, we can decrease the write-throughput requirements of the DynamoDB table.                                                                                                                                    |
| `input.maxRetries` (since 0.5.0)                    | Optional. Default value 10. Maximum number of retries for AWS SDK operations when reading from Kinesis.                                                                                                                                                                                                                                      |
| `input.apiCallAttemptTimeout` (since 0.5.0)         | Optional. Default value `15 seconds`. Maximum amount of time to wait for an individual Kinesis API call to complete.                                                                                                                                                                                                                         |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |
| `output.bad.maxRetries` (since 0.4.0)               | Optional. Default value 10. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                     |

**GCP:**

| Parameter                                 | Description                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                      | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                        |
| `input.durationPerAckExtension`           | Optional. Default value `15 seconds`. Pub/Sub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                               |
| `input.minRemainingAckDeadline`           | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `60 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `6 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.maxMessagesPerPull`                | Optional. Default value 1000. How many Pub/Sub messages to pull from the server in a single request.                                                                                                                                                                                                                                                                  |
| `input.debounceRequests`                  | Optional. Default value `100 millis`. Adds an artifical delay between consecutive requests to Pub/Sub for more messages. Under some circumstances, this was found to slightly alleviate a problem in which Pub/Sub might re-deliver the same messages multiple times.                                                                                                 |
| `input.retries.transientErrors.delay`     | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                            |
| `input.retries.transientErrors.attempts`  | Optional. Default value `10`. Max number of attempts, after which Enrich will crash and exit                                                                                                                                                                                                                                                                          |
| `output.bad.topic`                        | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                   |
| `output.bad.batchSize`                    | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                     |
| `output.bad.requestByteThreshold`         | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                        |
| `output.bad.maxRecordSize`                | Optional. Default value 9000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                         |
| `output.retries.transientErrors.delay`    | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                            |
| `output.retries.transientErrors.attempts` | Optional. Default value `10`. Max number of attempts, after which Enrich will crash and exit                                                                                                                                                                                                                                                                          |

**Azure:**

| Parameter                           | Description                                                                                                                                                                                  |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                   | Required. Name of the Kafka topic for the source of enriched events.                                                                                                                         |
| `input.bootstrapServers`            | Required. Hostname and port of Kafka bootstrap servers hosting the source of enriched events.                                                                                                |
| `input.consumerConf.*`              | Optional. A map of key/value pairs for [any standard Kafka consumer configuration option](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html).      |
| `input.debounceCommitOffsets`       | Optional. Default value 10 seconds. How frequently to commit our progress back to kafka. By increasing this value, we decrease the number of requests made to the kafka broker.              |
| `input.commitTimeout` (since 0.4.0) | Optional. Default value 15 seconds. The time to wait for offset commits to complete. If an offset commit doesn't complete within this time, a CommitTimeoutException will be raised instead. |
| `output.bad.topicName`              | Required. Name of the Kafka topic that will receive failed events.                                                                                                                           |
| `output.bad.bootstrapServers`       | Required. Hostname and port of Kafka bootstrap servers hosting the bad topic                                                                                                                 |
| `output.bad.producerConf.*`         | Optional. A map of key/value pairs for [any standard Kafka producer configuration option](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html).      |
| `output.bad.maxRecordSize`          | Optional. Default value 1000000. Any single failed event sent to Kafka should not exceed this size in bytes                                                                                  |

> **Info:** You can use the `input.consumerConf` and `output.bad.producerConf` options to configure authentication to Azure event hubs using SASL. For example:
>
> ```json
> "input.consumerConf": {
>
>     "security.protocol": "SASL_SSL"
>
>     "sasl.mechanism": "PLAIN"
>
>     "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=<PASSWORD>;"
>
> }
> ```

***

## Other configuration options

| Parameter                                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batching.maxBytes`                                  | Optional. Default value `16000000`. Events are uploaded to the Databricks volume when the batch reaches this size in bytes                                                                                                                                                                                                                                                                                                            |
| `batching.maxDelay`                                  | Optional. Default value `1 second`. Events are uploaded to the Databricks volume after a maximum of this duration, even if the `maxBytes` size has not been reached                                                                                                                                                                                                                                                                   |
| `batching.uploadParallelismFactor`                   | Optional. Default value 3. Controls how many batches can we send simultaneously over the network to Databricks. E.g. If there are 4 available processors, and `uploadParallelismFactor` is 3.5, then the loader sends up to 14 batches in parallel. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                  |
| `cpuParallelismFactor`                               | Optional. Default value 0.75. Controls how the app splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and `cpuParallelismFactor` is 0.75, then the loader processes 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                         |
| `retries.setupErrors.delay`                          | Optional. Default value `30 seconds`. Configures exponential backoff on errors related to how Databricks is set up for this loader. Examples include authentication errors and permissions errors. This class of errors are reported periodically to the monitoring webhook.                                                                                                                                                          |
| `retries.transientErrors.delay`                      | Optional. Default value `1 second`. Configures exponential backoff on errors that are likely to be transient. Examples include server errors and network errors.                                                                                                                                                                                                                                                                      |
| `retries.transientErrors.attempts`                   | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                                                                                                  |
| `skipSchemas`                                        | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`. A list of schemas that won't be loaded to Databricks. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                                                                                     |
| `decompression.maxBytesInBatch` (since 0.5.0)        | Optional. Default value `5242880` (5 MB).The loader automatically detects and decompresses zstd- or gzip-compressed source messages. Uncompressed messages are unaffected.The loader uses this as a cutoff when incrementally adding decompressed events to a batch, and emits the batch as soon as it reaches this size. This protects the loader's memory because a small compressed message can expand into a much larger payload. |
| `decompression.maxBytesSinglePayload` (since 0.5.0)  | Optional. Default value `10000000` (10 MB).For zstd- or gzip-compressed source messages, this is the maximum size of a single payload after decompression. The loader emits a [size violation failed event](/docs/api-reference/failed-events/#size-violation) for any payload that exceeds this size.                                                                                                                                |
| `exitOnMissingIgluSchema`                            | Optional. Default value `true`. Whether the loader should crash and exit if it fails to resolve an Iglu Schema. We recommend `true` because Snowplow enriched events have already passed validation, so a missing schema normally indicates an error that needs addressing. Change to `false` so events go the failed events stream instead of crashing the loader.                                                                   |
| `monitoring.metrics.statsd.hostname`                 | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.metrics.statsd.port`                     | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                                                                                          |
| `monitoring.metrics.statsd.tags.*`                   | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.metrics.statsd.period`                   | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.prefix`                   | Optional. Default `snowplow.databricks-loader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                                                                                               |
| `monitoring.metrics.prometheus.tags.*` (since 0.5.0) | Optional. A map of key/value pairs added as common labels on every Prometheus metric. The loader exposes these metrics at the `/metrics` endpoint on the health probe port.                                                                                                                                                                                                                                                           |
| `monitoring.webhook.endpoint`                        | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how Databricks is set up for this loader.                                                                                                                                                                                                                                                      |
| `monitoring.webhook.tags.*`                          | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.webhook.heartbeat.*`                     | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.sentry.dsn`                              | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.sentry.tags.*`                           | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                    |
| `telemetry.disable`                                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                             |
| `telemetry.userProvidedId`                           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                   |
| `http.client.maxConnectionsPerServer`                | Optional. Default value 4. Configures the internal HTTP client used for iglu resolver, alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time.                                                                                                                                                                                                                                           |

---

# Databricks Streaming Loader
> Load Snowplow events to Databricks with low latency using Lakeflow Declarative Pipelines and Unity Catalog volumes.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/

> **Info:** You will need a premium Databricks plan to use Lakeflow Declarative Pipelines.

The Databricks Streaming Loader is an application that integrates with a Databricks [Lakeflow Declarative Pipeline](https://docs.databricks.com/aws/en/dlt/) to load Snowplow events into Databricks with low latency.

**AWS:**

There are two parts to how the Databricks Streaming Loader works.

In the first part, you use Snowplow's Databricks Streaming Loader to push staging files into a [Unity Catalog volume](https://docs.databricks.com/aws/en/volumes/).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Databricks Streaming Loader</b>"}}
subgraph databricks [Databricks]
  volume[("<b>Volume</b>")]
end
stream-->loader-->|REST API|databricks
```

In the second part, you use a Databricks Lakeflow Declarative Pipeline to load the staging files into a Streaming Live Table.

```mermaid
flowchart LR
subgraph databricks [Databricks]
    direction LR
    volume[("<b>Volume</b>")]
    pipeline{{"<b>Lakeflow Declarative Pipeline</b>"}}
    table[("<b>Streaming Live Table</b>")]
    volume-->pipeline-->table
end
```

The Databricks Streaming Loader is published as a Docker image which you can run on any AWS VM.

```bash
docker pull snowplow/databricks-loader-kinesis:0.5.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting your Databricks credentials via environment variables, e.g. `DATABRICKS_CLIENT_SECRET`, so that you can refer to them in the config file.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

--env DATABRICKS_CLIENT_ID="${DATABRICKS_CLIENT_ID}" \

--env DATABRICKS_CLIENT_SECRET="${DATABRICKS_CLIENT_SECRET}" \

snowplow/databricks-loader-kinesis:0.5.0 \

--config=/myconfig/loader.hocon
```

Where `loader.hocon` is the loader's [configuration file](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/#configuring-the-loader) and `iglu.hocon` is the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

**GCP:**

There are two parts to how the Databricks Streaming Loader works.

In the first part, you use Snowplow's Databricks Streaming Loader to push staging files into a [Unity Catalog volume](https://docs.databricks.com/aws/en/volumes/).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Databricks Streaming Loader</b>"}}
subgraph databricks [Databricks]
  volume[("<b>Volume</b>")]
end
stream-->loader-->|REST API|databricks
```

In the second part, you use a Databricks Lakeflow Declarative Pipeline to load the staging files into a Streaming Live Table.

```mermaid
flowchart LR
subgraph databricks [Databricks]
    direction LR
    volume[("<b>Volume</b>")]
    pipeline{{"<b>Lakeflow Declarative Pipeline</b>"}}
    table[("<b>Streaming Live Table</b>")]
    volume-->pipeline-->table
end
```

The Databricks Streaming Loader is published as a Docker image which you can run on any GCP VM.

```bash
docker pull snowplow/databricks-loader-pubsub:0.5.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting your Databricks credentials via environment variables, e.g. `DATABRICKS_CLIENT_SECRET`, so that you can refer to them in the config file.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

--env DATABRICKS_CLIENT_ID="${DATABRICKS_CLIENT_ID}" \

--env DATABRICKS_CLIENT_SECRET="${DATABRICKS_CLIENT_SECRET}" \

snowplow/databricks-loader-pubsub:0.5.0 \

--config=/myconfig/loader.hocon
```

Where `loader.hocon` is the loader's [configuration file](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/#configuring-the-loader) and `iglu.hocon` is the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

**Azure:**

There are two parts to how the Databricks Streaming Loader works.

In the first part, you use Snowplow's Databricks Streaming Loader to push staging files into a [Unity Catalog volume](https://docs.databricks.com/aws/en/volumes/).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Databricks Streaming Loader</b>"}}
subgraph databricks [Databricks]
  volume[("<b>Volume</b>")]
end
stream-->loader-->|REST API|databricks
```

In the second part, you use a Databricks Lakeflow Declarative Pipeline to load the staging files into a Streaming Live Table.

```mermaid
flowchart LR
subgraph databricks [Databricks]
    direction LR
    volume[("<b>Volume</b>")]
    pipeline{{"<b>Lakeflow Declarative Pipeline</b>"}}
    table[("<b>Streaming Live Table</b>")]
    volume-->pipeline-->table
end
```

The Databricks Streaming Loader is published as a Docker image which you can run on any Azure VM.

```bash
docker pull snowplow/databricks-loader-kafka:0.5.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting your Databricks credentials via environment variables, e.g. `DATABRICKS_CLIENT_SECRET`, so that you can refer to them in the config file.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

--env DATABRICKS_CLIENT_ID="${DATABRICKS_CLIENT_ID}" \

--env DATABRICKS_CLIENT_SECRET="${DATABRICKS_CLIENT_SECRET}" \

snowplow/databricks-loader-kafka:0.5.0 \

--config=/myconfig/loader.hocon
```

Where `loader.hocon` is the loader's [configuration file](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/#configuring-the-loader) and `iglu.hocon` is the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

***

> **Tip:** For more information on how events are stored in Databricks, check the [mapping between Snowplow schemas and the corresponding Databricks column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=databricks).

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/databricks-loader/blob/develop/config/config.kinesis.minimal.hocon&#xA;>)

**GCP:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/databricks-loader/blob/develop/config/config.pubsub.minimal.hocon&#xA;>)

**Azure:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/databricks-loader/blob/develop/config/config.kafka.minimal.hocon&#xA;>)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/configuration-reference/) for all possible configuration parameters.

### Iglu

The Databricks Streaming Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

## Configuring the Databricks Lakeflow Declarative Pipeline

Create a Pipeline in your Databricks workspace, and copy the following SQL into the associated `.sql` file:

```sql
CREATE STREAMING LIVE TABLE events

CLUSTER BY (load_tstamp, event_name)

TBLPROPERTIES (

  'delta.dataSkippingStatsColumns' =

      'load_tstamp,collector_tstamp,derived_tstamp,dvce_created_tstamp,true_tstamp,event_name'

)

AS SELECT

  *,

  current_timestamp() as load_tstamp

FROM cloud_files(

  "/Volumes/<CATALOG_NAME>/<VOLUME_NAME>/<SCHEMA_NAME>/events",

  "parquet",

  map(

    "cloudfiles.inferColumnTypes", "false",

    "cloudfiles.includeExistingFiles", "false", -- set to true to load files already present in the volume

    "cloudfiles.schemaEvolutionMode", "addNewColumns",

    "cloudfiles.partitionColumns", "",

    "cloudfiles.useManagedFileEvents", "true",

    "datetimeRebaseMode", "CORRECTED",

    "int96RebaseMode", "CORRECTED",

    "mergeSchema", "true"

  )

)
```

Replace `/Volumes/<CATALOG_NAME>/<VOLUME_NAME>/<SCHEMA_NAME>/events` with the correct path to your volume.

Note that the volume must be an [external volume](https://docs.databricks.com/aws/en/volumes/) in to order to use the cloud files option `cloudfiles.useManagedFileEvents`, which is highly recommended for this integration.

## Metrics

The Databricks Streaming Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:

| Metric               | Definition                                                                                                                                                                |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_good`        | A count of events that are successfully written to the Databricks volume.                                                                                                 |
| `events_bad`         | A count of failed events that could not be loaded, and were instead sent to the bad output stream.                                                                        |
| `latency_millis`     | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader.                             |
| `e2e_latency_millis` | The end-to-end latency of the snowplow pipeline. The time in milliseconds from when an event was received by the collector, until it is written to the Databricks volume. |

See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/configuration-reference/) for how to configure the StatsD receiver.

**Telemetry notice**

By default, Snowplow collects telemetry data for Databricks Streaming Loader (since version 0.1.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Elasticsearch Loader
> Load Snowplow enriched and bad events from a Kinesis stream into Elasticsearch or OpenSearch clusters.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/elasticsearch/

If you are using [Enrich](/docs/api-reference/enrichment-components/) to write enriched Snowplow events to one stream and bad events to another, you can use the Elasticsearch Loader to read events from a Kinesis stream and write them to [Elasticsearch](http://www.elasticsearch.org/overview/) or [OpenSearch](https://opensearch.org/).

> **Note:** We only offer this loader on AWS or as part of [Snowplow Mini](/docs/api-reference/snowplow-mini/).

## What the data looks like

There are a few changes compared to the [standard structure of Snowplow data](/docs/fundamentals/canonical-event/).

### Boolean fields reformatted

All boolean fields like `br_features_java` are normally either `"0"` or `"1"`. In Elasticsearch, these values are converted to `false` and `true`.

### New `geo_location` field

The `geo_latitude` and `geo_longitude` fields are combined into a single `geo_location` field of Elasticsearch's ["geo\_point" type](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html).

### Self-describing events

Each [self-describing event](/docs/fundamentals/events/#self-describing-events) gets its own field (same [naming rules](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=snowflake#location) as for Snowflake). For example:

```json
{

  "unstruct_com_snowplowanalytics_snowplow_link_click_1": {

    "targetUrl": "http://snowplow.io",

    "elementId": "action",

    "elementClasses": [],

    "elementTarget": ""

  }

}
```

### Entities

Each [entity](/docs/fundamentals/entities/) type attached to the event gets its own field (same [naming rules](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=snowflake#location) as for Snowflake). The field contains an array with the data for all entities of the given type. For example:

```json
{

  "contexts_com_acme_user_1": [

    {

      "name": "Alice"

    }

  ],

  "contexts_com_acme_product_1": [

    {

      "name": "Apple"

    },

    {

      "name": "Orange"

    }

  ]

}
```

## Setup guide

### Configuring Elasticsearch

#### Getting started

First, install and set up Elasticsearch. For more information, see the [installation guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html) for installation information and [Supported versions of OpenSearch and Elasticsearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version) for the latest information of ElasticSearch/OpenSearch supported versions by AWS.

> **Note:** The Elasticsearch Loader supports Elasticsearch 6.x, 7.x, 8.x, and 9.x. It also supports OpenSearch 1.x, 2.x, and 3.x.

#### Raising the file limit

Elasticsearch keeps a lot of files open simultaneously, so you will need to increase the maximum number of files a user can have open. To do this:

```bash
sudo vim /etc/security/limits.conf
```

Append the following lines to the file:

```bash
{{USERNAME}} soft nofile 32000

{{USERNAME}} hard nofile 32000
```

Where `{{USERNAME}}` is the name of the user running Elasticsearch. You will need to logout and restart Elasticsearch before the new file limit takes effect.

To check that this new limit has taken effect you can run the following command from the terminal:

```bash
curl localhost:9200/_nodes/process?pretty
```

If the `max_file_descriptors` equals 32000 it is running with the new limit.

#### Defining the mapping

Use the following request to create the mapping with Elasticsearch 7+:

```bash
curl -XPUT 'http://localhost:9200/snowplow' -d '{

  "settings": {

      "analysis": {

          "analyzer": {

              "default": {

                  "type": "keyword"

              }

          }

      }

  },

  "mappings": {

      "properties": {

          "geo_location": {

              "type": "geo_point"

          }

      }

  }

}'
```

Note that Elasticsearch 7+ [no longer uses mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html). If you have an older version, you might need to include mapping types in the above snippet.

This initialization sets the default analyzer to "keyword". This means that string fields will not be split into separate tokens for the purposes of searching. This saves space and ensures that URL fields are handled correctly.

If you want to tokenize specific string fields, you can change the "properties" field in the mapping like this:

```bash
curl -XPUT 'http://localhost:9200/snowplow' -d '{

  "settings": {

      "analysis": {

          "analyzer": {

              "default": {

                  "type": "keyword"

              }

          }

      }

  },

  "mappings": {

      "properties": {

          "geo_location": {

              "type": "geo_point"

          },

          "field_to_tokenize": {

              "type": "string",

              "analyzer": "english"

          }

      }

  }

}'
```

### Installing the Elasticsearch Loader

The Elasticsearch Loader is published on Docker Hub:

```bash
docker pull snowplow/elasticsearch-loader:3.0.1
```

The container can be run with the following command:

```bash
docker run \

-v /path/to/config.hocon:/snowplow/config.hocon \

snowplow/elasticsearch-loader:3.0.1 \

--config /snowplow/config.hocon
```

### Configure the Elasticsearch Loader

The loader is configured using a HOCON file. You can find a minimal example and a full reference example in the [config directory](https://github.com/snowplow/snowplow-elasticsearch-loader/tree/master/config).

#### License acceptance

The loader requires explicit acceptance of the [Snowplow Limited Use License Agreement](https://docs.snowplow.io/limited-use-license-1.1):

```hocon
"license": {

  "accept": true

}
```

#### Input

The loader reads from a Kinesis stream. The following fields configure the input:

| Name                                                | Description                                                                                                                                                                                                                                 |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream to read from.                                                                                                                                                                                          |
| `input.appName`                                     | Optional. Name used for the KCL DynamoDB lease table. Default: `"snowplow-elasticsearch-loader"`.                                                                                                                                           |
| `input.initialPosition.type`                        | Optional. Where to start reading the first time the app runs. Options: `"TRIM_HORIZON"` (oldest), `"LATEST"` (newest), or `"AT_TIMESTAMP"`. Default: `"TRIM_HORIZON"`. On subsequent runs, the app always resumes from the last checkpoint. |
| `input.initialPosition.timestamp`                   | Required when `initialPosition.type` is `"AT_TIMESTAMP"`. Timestamp to start reading from, e.g. `"2023-01-01T00:00:00Z"`.                                                                                                                   |
| `input.retrievalMode.type`                          | Optional. How the KCL fetches events. Options: `"Polling"` or `"FanOut"` (Kinesis Enhanced Fan-Out). Default: `"Polling"`.                                                                                                                  |
| `input.retrievalMode.maxRecords`                    | Optional. Used when `retrievalMode.type` is `"Polling"`. Maximum number of events per poll request. Default: `750`.                                                                                                                         |
| `input.retrievalMode.idleTimeBetweenReads`          | Optional. Used when `retrievalMode.type` is `"Polling"`. Idle time between `GetRecords` requests. Default: `"1500 millis"`.                                                                                                                 |
| `input.workerIdentifier`                            | Optional. Name of this KCL worker in the DynamoDB lease table. Default: `${HOSTNAME}`.                                                                                                                                                      |
| `input.leaseDuration`                               | Optional. Duration of shard leases. Workers must refresh leases in DynamoDB before this expires. Default: `"10 seconds"`.                                                                                                                   |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Controls how many leases can be stolen at once, as a multiple of available processors. Default: `2.0`.                                                                                                                            |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Minimum backoff when DynamoDB provisioned throughput limits are hit. Default: `"100 millis"`.                                                                                                                                     |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Maximum backoff when DynamoDB provisioned throughput limits are hit. Default: `"1 second"`.                                                                                                                                       |
| `input.debounceCheckpoints`                         | Optional. How often to checkpoint progress to DynamoDB. Default: `"10 seconds"`.                                                                                                                                                            |
| `input.maxRetries`                                  | Optional. Maximum number of retries for Kinesis API operations. Default: `10`.                                                                                                                                                              |
| `input.apiCallAttemptTimeout`                       | Optional. Maximum time for a single AWS API call attempt. Default: `"15 seconds"`.                                                                                                                                                          |

#### Output: good events

Good events are written to Elasticsearch. The following fields configure the Elasticsearch output:

| Name                                     | Description                                                                                                                                                                                                                                     |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.url`                        | Required. URL of the Elasticsearch cluster, including scheme and port. Example: `"http://localhost:9200"`.                                                                                                                                      |
| `output.good.index`                      | Required. Name of the Elasticsearch index to write events into.                                                                                                                                                                                 |
| `output.good.auth.type`                  | Required. Authentication method. Options: `"NoAuth"`, `"Basic"`, or `"AWSSigning"`.                                                                                                                                                             |
| `output.good.auth.username`              | Required when `auth.type` is `"Basic"`. HTTP Basic Auth username.                                                                                                                                                                               |
| `output.good.auth.password`              | Required when `auth.type` is `"Basic"`. HTTP Basic Auth password.                                                                                                                                                                               |
| `output.good.auth.region`                | Required when `auth.type` is `"AWSSigning"`. AWS region of the OpenSearch Service domain.                                                                                                                                                       |
| `output.good.auth.serviceSigningName`    | Required when `auth.type` is `"AWSSigning"`. AWS service name for SigV4 signing. Use `"es"` for OpenSearch Service or `"aoss"` for OpenSearch Serverless.                                                                                       |
| `output.good.documentType`               | Optional. Elasticsearch document type. Only required for Elasticsearch 6.x compatibility.                                                                                                                                                       |
| `output.good.sharding.dateFormat`        | Optional. Date format to append to the index name for time-partitioned indices, e.g. `"yyyy-MM-dd"`.                                                                                                                                            |
| `output.good.sharding.dateField`         | Required when `sharding` is set. Timestamp field to extract the date from for index sharding. Must be one of: `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`, `dvce_sent_tstamp`, `etl_tstamp`, `refr_dvce_tstamp`, `true_tstamp`. |
| `output.good.indexTimeout`               | Optional. Timeout passed to Elasticsearch for each bulk request. Default: `"1 minute"`.                                                                                                                                                         |
| `output.good.additionalBadRowErrorTypes` | Optional. Additional Elasticsearch error types to treat as permanent bad rows instead of retrying. By default, `mapper_parsing_exception` and `document_parsing_exception` are treated as bad rows.                                             |

#### Output: bad rows

Events that cannot be written to Elasticsearch are sent to a Kinesis stream as bad rows:

| Name                                           | Description                                                                                                                     |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `output.bad.streamName`                        | Required. Name of the Kinesis stream for bad rows.                                                                              |
| `output.bad.maxRecordSize`                     | Optional. Maximum record size in bytes. Records exceeding this are replaced with a `SizeViolation` bad row. Default: `1000000`. |
| `output.bad.throttledBackoffPolicy.minBackoff` | Optional. Minimum backoff when Kinesis write throughput limits are exceeded. Default: `"100 milliseconds"`.                     |
| `output.bad.throttledBackoffPolicy.maxBackoff` | Optional. Maximum backoff when Kinesis write throughput limits are exceeded. Default: `"1 second"`.                             |
| `output.bad.recordLimit`                       | Optional. Maximum number of records per `PutRecords` request. Default: `500`.                                                   |
| `output.bad.byteLimit`                         | Optional. Maximum number of bytes per `PutRecords` request. Default: `5242880`.                                                 |
| `output.bad.maxRetries`                        | Optional. Maximum number of retries for Kinesis write operations. Default: `10`.                                                |

#### Purpose

| Name      | Description                                                                                                                                                         |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `purpose` | Required. Type of events to process. Options: `"ENRICHED_EVENTS"` for Snowplow enriched events, `"BAD_ROWS"` for Snowplow bad rows, or `"JSON"` for arbitrary JSON. |

#### Batching

The loader accumulates events into batches before sending them to Elasticsearch. A batch is flushed when the first condition is met:

| Name                | Description                                                                              |
| ------------------- | ---------------------------------------------------------------------------------------- |
| `batching.maxBytes` | Optional. Flush after this many source bytes have been accumulated. Default: `10000000`. |
| `batching.maxDelay` | Optional. Flush after this delay has elapsed. Default: `"1 second"`.                     |

#### Retries

| Name                               | Description                                                                                       |
| ---------------------------------- | ------------------------------------------------------------------------------------------------- |
| `retries.transientErrors.delay`    | Optional. Delay between retry attempts for transient Elasticsearch errors. Default: `"1 second"`. |
| `retries.transientErrors.attempts` | Optional. Maximum number of retry attempts before treating the batch as failed. Default: `5`.     |

#### Decompression

The loader automatically detects and decompresses zstd- or gzip-compressed Kinesis messages. Uncompressed messages are unaffected.

| Name                                  | Description                                                                                                                                                 |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `decompression.maxBytesInBatch`       | Optional. Maximum total decompressed bytes per batch. Protects memory when a single compressed message expands into many large records. Default: `5242880`. |
| `decompression.maxBytesSinglePayload` | Optional. Maximum size of a single decompressed record in bytes. Records exceeding this limit are dropped and emitted as bad rows. Default: `10000000`.     |

#### Parallelism

| Name                      | Description                                                                                                                                                     |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpuParallelismFactor`    | Optional. Controls how the app splits the workload into concurrent batches for parsing and transformation, as a multiple of available processors. Default: `1`. |
| `uploadParallelismFactor` | Optional. Controls how many Elasticsearch bulk upload jobs can run in parallel, as a multiple of available processors. Default: `4`.                            |

#### Monitoring

The loader exposes runtime metrics and health information through several optional integrations:

| Name                                      | Description                                                                                                                                                                                                 |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `monitoring.metrics.statsd.hostname`      | Optional. Hostname of the StatsD server to send metrics to.                                                                                                                                                 |
| `monitoring.metrics.statsd.port`          | Optional. Port of the StatsD server. Default: `8125`.                                                                                                                                                       |
| `monitoring.metrics.statsd.tags`          | Optional. Map of key/value pairs sent along with each metric.                                                                                                                                               |
| `monitoring.metrics.statsd.period`        | Optional. How often to report metrics. Default: `"1 minute"`.                                                                                                                                               |
| `monitoring.metrics.statsd.prefix`        | Optional. Prefix for metric names.                                                                                                                                                                          |
| `monitoring.metrics.prometheus.tags`      | Optional. Map of key/value pairs used as common labels on all Prometheus metrics. The loader exposes these metrics at the `/metrics` endpoint on the health probe port (see `monitoring.healthProbe.port`). |
| `monitoring.sentry.dsn`                   | Optional. Sentry DSN for reporting unexpected runtime exceptions.                                                                                                                                           |
| `monitoring.sentry.tags`                  | Optional. Map of key/value pairs included as tags on Sentry events.                                                                                                                                         |
| `monitoring.healthProbe.port`             | Optional. Port for the HTTP health probe server. Returns `200 OK` when healthy.                                                                                                                             |
| `monitoring.healthProbe.unhealthyLatency` | Optional. The health probe becomes unhealthy if any received event has not been fully processed before this cutoff time. Default: `"2 minutes"`.                                                            |

#### Telemetry

| Name                       | Description                                                                    |
| -------------------------- | ------------------------------------------------------------------------------ |
| `telemetry.disable`        | Optional. Set to `true` to disable telemetry. Default: `false`.                |
| `telemetry.userProvidedId` | Optional. Identifier to tie events together across modules and infrastructure. |

## Check document count

To check the number of documents in an Elasticsearch or OpenSearch cluster, use the [Count API](https://docs.opensearch.org/latest/api-reference/search-apis/count/) provided by Elasticsearch/OpenSearch. For example, to get the total number of documents in the cluster, use `GET _count`.

---

# Elasticsearch Loader 3.0.x upgrade guide
> Upgrade guide for Snowplow Elasticsearch Loader 3.0.x covering common-streams refactoring, configuration changes, and removed features.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/elasticsearch/upgrade-guides/3-0-x-upgrade-guide/

In version 3.0.0, the Elasticsearch Loader is rewritten to use [common-streams](https://github.com/snowplow-incubator/common-streams) libraries under the hood. [common-streams](https://github.com/snowplow-incubator/common-streams) is the collection of libraries that contains streaming-related constructs commonly used across many Snowplow streaming applications.

This is a breaking change: the configuration format has changed significantly. You will need to migrate your configuration file before upgrading.

## Removed features

The following features from 2.x are no longer available in 3.0.0.

### Remove NSQ support

The Elasticsearch Loader no longer supports NSQ as an input or output. Only Kinesis is supported. If you are running the loader with NSQ, you will need to migrate to Kinesis before upgrading.

### Remove CloudWatch metrics

The `monitoring.metrics.cloudWatch` config option is removed. Metrics are now reported via [StatsD](/docs/api-reference/loaders-storage-targets/elasticsearch/#monitoring) or exposed via a [Prometheus `/metrics` endpoint](/docs/api-reference/loaders-storage-targets/elasticsearch/#monitoring).

### Remove Snowplow monitoring

The `monitoring.snowplow` section (collector URI and app ID for self-monitoring events) is removed. There is no replacement.

## Configuration changes

The configuration format has changed significantly. The sections below describe each change and how to migrate.

### Accept the license

A new top-level `license` section is required:

```hocon
"license": {

  "accept": true

}
```

### Configure the input

The `input.type` field is removed. Kinesis is the only supported input and no longer needs to be specified.

`input.initialPosition` changes from a plain string to an object:

```hocon
# Before

"input": {

  "initialPosition": "TRIM_HORIZON"

  "initialTimestamp": "2023-01-01T00:00:00Z"  # only used for AT_TIMESTAMP

}



# After

"input": {

  "initialPosition": {

    "type": "TRIM_HORIZON"

    # "timestamp": "2023-01-01T00:00:00Z"  # only required for AT_TIMESTAMP

  }

}
```

`input.maxRecords` moves into the new `input.retrievalMode` object:

```hocon
# Before

"input": {

  "maxRecords": 10000

}



# After

"input": {

  "retrievalMode": {

    "type": "Polling"

    "maxRecords": 750

  }

}
```

Kinesis Enhanced Fan-Out is now supported by setting `retrievalMode.type` to `"FanOut"`.

The `input.region` field is removed. Region is resolved automatically via the [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).

The `input.buffer` section is removed. Batching is now controlled by the top-level [`batching`](/docs/api-reference/loaders-storage-targets/elasticsearch/#batching) section.

### Output: good events

The `output.good.type` field is removed. Elasticsearch is the only supported output.

`output.good.client.endpoint`, `output.good.client.port`, and `output.good.client.ssl` are replaced by a single `output.good.url` field that includes the scheme, host, and port:

```hocon
# Before

"output": {

  "good": {

    "client": {

      "endpoint": "localhost"

      "port": 9200

      "ssl": false

    }

  }

}



# After

"output": {

  "good": {

    "url": "http://localhost:9200"

  }

}
```

Authentication is now configured via a unified `output.good.auth` object, replacing the separate `output.good.client.username`/`password` and `output.good.aws.signing`/`region` fields:

```hocon
# Before (HTTP Basic Auth)

"output": {

  "good": {

    "client": {

      "username": "elastic"

      "password": "changeme"

    }

  }

}



# After (HTTP Basic Auth)

"output": {

  "good": {

    "auth": {

      "type": "Basic"

      "username": "elastic"

      "password": "changeme"

    }

  }

}
```

```hocon
# Before (AWS SigV4 signing)

"output": {

  "good": {

    "aws": {

      "signing": true

      "region": "eu-west-1"

    }

  }

}



# After (AWS SigV4 signing)

"output": {

  "good": {

    "auth": {

      "type": "AWSSigning"

      "region": "eu-west-1"

      "serviceSigningName": "es"  # use "aoss" for OpenSearch Serverless

    }

  }

}
```

`output.good.cluster.index` and `output.good.cluster.documentType` are moved up one level:

```hocon
# Before

"output": {

  "good": {

    "cluster": {

      "index": "snowplow"

      "documentType": "_doc"

    }

  }

}



# After

"output": {

  "good": {

    "index": "snowplow"

    # "documentType": "_doc"  # only needed for ES 6.x

  }

}
```

Index sharding fields move from `output.good.client` into a dedicated `output.good.sharding` object:

```hocon
# Before

"output": {

  "good": {

    "client": {

      "shardDateFormat": "yyyy-MM-dd"

      "shardDateField": "derived_tstamp"

    }

  }

}



# After

"output": {

  "good": {

    "sharding": {

      "dateFormat": "yyyy-MM-dd"

      "dateField": "derived_tstamp"

    }

  }

}
```

`output.good.client.maxRetries` is replaced by `retries.transientErrors.attempts` in the top-level [`retries`](/docs/api-reference/loaders-storage-targets/elasticsearch/#retries) section.

`output.good.chunk.byteLimit` and `output.good.chunk.recordLimit` are removed. Batching is now controlled by the top-level [`batching`](/docs/api-reference/loaders-storage-targets/elasticsearch/#batching) section.

### Output: bad rows

`output.bad.type` is removed. Bad rows are always written to Kinesis.

`output.bad.region` is removed. Region is resolved automatically.

### Monitoring

`monitoring.snowplow` and `monitoring.metrics.cloudWatch` are removed. Replace with the new monitoring options:

```hocon
# After

"monitoring": {

  "metrics": {

    "statsd": {

      "hostname": "127.0.0.1"

      "port": 8125

      "period": "1 minute"

      "prefix": "snowplow.elasticsearch.loader"

    }

    # or expose a Prometheus /metrics endpoint:

    # "prometheus": {}

  }

  "healthProbe": {

    "port": 8000

  }

}
```

## New features

Version 3.0.0 introduces the following new capabilities.

### Decompression

The loader now automatically decompresses zstd- and gzip-compressed Kinesis messages. No configuration is required to enable this. See the [`decompression`](/docs/api-reference/loaders-storage-targets/elasticsearch/#decompression) section for optional tuning.

### Sentry integration

Unexpected runtime exceptions can now be reported to Sentry via `monitoring.sentry.dsn`.

## Metrics changes

The following metrics are available in 3.0.x:

| Metric                         | Description                                                                                             |
| ------------------------------ | ------------------------------------------------------------------------------------------------------- |
| `events_good`                  | Count of events successfully written to Elasticsearch.                                                  |
| `events_bad`                   | Count of events sent to the bad rows stream.                                                            |
| `latency_millis`               | Delay between the input record being written to Kinesis and the loader starting to process it.          |
| `e2e_latency_millis`           | End-to-end latency from the input record being written to Kinesis to it being written to Elasticsearch. |
| `elasticsearch_latency_millis` | Time taken for Elasticsearch bulk requests to complete.                                                 |

## Supported Elasticsearch and OpenSearch versions

Version 3.0.0 adds support for Elasticsearch 8.x and 9.x, and OpenSearch 3.x. The full supported range is now Elasticsearch 6.x–9.x and OpenSearch 1.x–3.x.

---

# Upgrade guides for Elasticsearch Loader
> Step-by-step guides for upgrading Snowplow Elasticsearch Loader to newer versions with breaking changes and migration instructions.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/elasticsearch/upgrade-guides/

This section contains information to help you upgrade to newer versions of the Elasticsearch Loader.

## [3.0.x upgrade guide](/docs/api-reference/loaders-storage-targets/elasticsearch/upgrade-guides/3-0-x-upgrade-guide/)

[Upgrade guide for Snowplow Elasticsearch Loader 3.0.x covering common-streams refactoring, configuration changes, and removed features.](/docs/api-reference/loaders-storage-targets/elasticsearch/upgrade-guides/3-0-x-upgrade-guide/)

---

# Google Cloud Storage Loader
> Archive Snowplow events from Pub/Sub to Google Cloud Storage buckets using Dataflow with windowing and partitioning.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/

[Cloud Storage Loader](https://github.com/snowplow-incubator/snowplow-google-cloud-storage-loader/) is a [Dataflow](https://cloud.google.com/dataflow/) job which dumps event from an input [PubSub](https://cloud.google.com/pubsub/) subscription into a [Cloud Storage](https://cloud.google.com/storage/) bucket.

Cloud Storage loader is built on top of [Apache Beam](https://beam.apache.org/) and its Scala wrapper [SCIO](https://github.com/spotify/scio).

## Running

Cloud Storage Loader comes both as a Docker image and a ZIP archive.

### Docker

Docker image can be found on [Docker Hub](https://hub.docker.com/r/snowplow/snowplow-google-cloud-storage-loader).

A container can be run as follows:

```bash
docker run \

-v $PWD/config:/snowplow/config \ # if running outside GCP

-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/config/credentials.json \ # if running outside GCP

snowplow/snowplow-google-cloud-storage-loader:0.5.7 \

--runner=DataFlowRunner \

--jobName=[JOB-NAME] \

--project=[PROJECT] \

--streaming=true \

--workerZone=[ZONE] \

--inputSubscription=projects/[PROJECT]/subscriptions/[SUBSCRIPTION] \

--outputDirectory=gs://[BUCKET]/YYYY/MM/dd/HH/ \ # partitions by date

--outputFilenamePrefix=output \ # optional

--shardTemplate=-W-P-SSSSS-of-NNNNN \ # optional

--outputFilenameSuffix=.txt \ # optional

--windowDuration=5 \ # optional, in minutes

--compression=none \ # optional, gzip, bz2 or none

--numShards=1 # optional
```

To display the help message:

```bash
docker run snowplow/snowplow-google-cloud-storage-loader:0.5.7 \

--help
```

To display documentation about Cloud Storage Loader-specific options:

```bash
docker run snowplow/snowplow-google-cloud-storage-loader:0.5.7 \

--help=com.snowplowanalytics.storage.googlecloudstorage.loader.Options
```

### ZIP archive

Archive is hosted on GitHub at this URI:

```bash
https://github.com/snowplow-incubator/snowplow-google-cloud-storage-loader/releases/download/0.5.7/snowplow-google-cloud-storage-loader-0.5.7.zip
```

Once unzipped the artifact can be run as follows:

```bash
./bin/snowplow-google-cloud-storage-loader \

--runner=DataFlowRunner \

--project=[PROJECT] \

--streaming=true \

--workerZone=[ZONE] \

--inputSubscription=projects/[PROJECT]/subscriptions/[SUBSCRIPTION] \

--outputDirectory=gs://[BUCKET]/YYYY/MM/dd/HH/ \ # partitions by date

--outputFilenamePrefix=output \ # optional

--shardTemplate=-W-P-SSSSS-of-NNNNN \ # optional

--outputFilenameSuffix=.txt \ # optional

--windowDuration=5 \ # optional, in minutes

--compression=none \ # optional, gzip, bz2 or none

--numShards=1 # optional
```

To display the help message:

```bash
./bin/snowplow-google-cloud-storage-loader --help
```

To display documentation about Cloud Storage Loader-specific options:

```bash
./bin/snowplow-google-cloud-storage-loader --help=com.snowplowanalytics.storage.googlecloudstorage.loader.Options
```

## Configuration

### Cloud Storage Loader specific options

- `--inputSubscription=String` The Cloud Pub/Sub subscription to read from, formatted like projects/\[PROJECT]/subscriptions/\[SUB]. Required.
- `--outputDirectory=gs://[BUCKET]/` The Cloud Storage directory to output files to, ending in /. Required.
- `--outputFilenamePrefix=String` The prefix for output files. Default: output. Optional.
- `--shardTemplate=String` A valid shard template as described [here](https://javadoc.io/static/com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-all/1.7.0/com/google/cloud/dataflow/sdk/io/ShardNameTemplate.html), which will be part of the filenames. Default: `-W-P-SSSSS-of-NNNNN`. Optional.
- `--outputFilenameSuffix=String` The suffix for output files. Default: .txt. Optional.
- `--windowDuration=Int` The window duration in minutes. Default: 5. Optional.
- `--compression=String` The compression used (gzip, bz2 or none). Note that bz2 can't be loaded into BigQuery. Default: no compression. Optional.
- `--numShards=Int` The maximum number of output shards produced when writing. Default: 1. Optional.
- `--dateFormat=YYYY/MM/dd/HH/` A date format string used for partitioning via date in `outputDirectory` and `partitionedOutputDirectory`. Default: `YYYY/MM/dd/HH/`. Optional. For example, the date format `YYYY/MM/dd/HH/` would produce a directory structure like this:
  ```bash
  gs://bucket/

    └── 2022

      └── 12

        └── 15

          ├── ...

          ├── 18

          ├── 19

          ├── 20

          └── ...
  ```
- `--partitionedOutputDirectory=gs://[BUCKET]/` The Cloud Storage directory to output files to, partitioned by schema, ending with /. Unpartitioned data will be sent to `outputDirectory`. Optional.

### Dataflow options

To run the Cloud Storage Loader on Dataflow, it is also necessary to specify additional configuration options. None of these options have default values, and they are all required.

- `--runner=DataFlowRunner` Passing the string `DataFlowRunner` specifies that we want to run on Dataflow.
- `--jobName=[NAME]` Specify a name for your Dataflow job that will be created.
- `--project=[PROJECT]` The name of your GCP project.
- `--streaming=true` Pass `true` to notify Dataflow that we're running a streaming application.
- `--workerZone=[ZONE]` The [zone](https://cloud.google.com/compute/docs/regions-zones) where the Dataflow nodes (effectively [GCP Compute Engine](https://cloud.google.com/compute/) nodes) will be launched.
- `--region=[REGION]` The [region](https://cloud.google.com/compute/docs/regions-zones) where the Dataflow job will be launched.
- `--gcpTempLocation=gs://[BUCKET]/` The GCS bucket where temporary files necessary to run the job (e.g. JARs) will be stored.

The list of all the options can be found at [the Dataflow documentation](https://docs.cloud.google.com/dataflow/docs/reference/pipeline-options?hl=en).

---

# Warehouse and lake data loaders
> Load Snowplow enriched events into data warehouses and lakes including BigQuery, Redshift, Snowflake, Databricks, and S3.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/

Snowplow provides loader applications for several different warehouse and lakes, across different clouds.

Choose a loader based on your use case and data needs.

---

# Lake Loader configuration reference
> Configure Lake Loader for Delta Lake and Iceberg tables with Kinesis, Pub/Sub, and Kafka stream settings for data lakes.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/

The configuration reference in this page is written for Lake Loader `0.11.0`

### License

The Lake Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {

  "accept": true

}
```

### Table configuration

**Delta Lake:**

| Parameter                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.location`               | Required, e.g. `gs://mybucket/events`. URI of the bucket location to which to write Snowplow enriched events in Delta format. The URI should start with the following prefix:* `s3a://` on AWS
* `gs://` on GCP
* `abfs://` on Azure                                                                                                                                                                                                                                               |
| `output.good.deltaTableProperties.*` | Optional. A map of key/value strings corresponding to Delta's table properties. These can be anything [from the Delta table properties documentation](https://docs.delta.io/latest/table-properties.html). The default properties include configuring Delta's [data skipping feature](https://docs.delta.io/latest/optimizations-oss.html#data-skipping) for the important Snowplow timestamp columns: `load_tstamp`, `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`. |

**Iceberg / Glue:**

| Parameter                              | Description                                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `output.good.type`                     | Required, set this to `Iceberg`                                                                                                                                                                                                                                                                                                                                                                                          |
| `output.good.catalog.type`             | Required, set this to `Glue`                                                                                                                                                                                                                                                                                                                                                                                             |
| `output.good.location`                 | Optional, e.g. `s3a://mybucket/events`. URI of the bucket location to which to write Snowplow enriched events in Iceberg format. The URI should start with `s3a://`. If not provided, the catalog's default warehouse location will be used.                                                                                                                                                                             |
| `output.good.database`                 | Required. Name of the database in the Glue catalog                                                                                                                                                                                                                                                                                                                                                                       |
| `output.good.table`                    | Required. The name of the table in the Glue database                                                                                                                                                                                                                                                                                                                                                                     |
| `output.good.icebergTableProperties.*` | Optional. A map of key/value strings corresponding to Iceberg's table properties. These can be anything [from the Iceberg table properties documentation](https://iceberg.apache.org/docs/latest/configuration/). The default properties include configuring Iceberg's column-level statistics for the important Snowplow timestamp columns: `load_tstamp`, `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`. |
| `output.good.catalog.options.*`        | Optional. A map of key/value strings which are passed to the catalog configuration. These can be anything [from the Iceberg catalog documentation](https://iceberg.apache.org/docs/latest/aws/) e.g. `"glue.id": "1234567"`                                                                                                                                                                                              |

**Iceberg / REST:**

> **Note:** The REST catalog integration has been tested with Snowflake Open Catalog.

| Parameter                              | Description                                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `output.good.type`                     | Required, set this to `Iceberg`                                                                                                                                                                                                                                                                                                                                                                                          |
| `output.good.catalog.type`             | Required, set this to `Rest`                                                                                                                                                                                                                                                                                                                                                                                             |
| `output.good.catalog.uri`              | Required. URI of the REST catalog server, e.g. `http://localhost:8080`                                                                                                                                                                                                                                                                                                                                                   |
| `output.good.catalog.name`             | Required. Name of the catalog                                                                                                                                                                                                                                                                                                                                                                                            |
| `output.good.location`                 | Optional. URI of the bucket location to which to write Snowplow enriched events in Iceberg format. The URI should start with `s3a://`, `gs://`, or `abfs://` depending on your cloud provider. If not provided, the catalog's default warehouse location will be used.                                                                                                                                                   |
| `output.good.database`                 | Required. Name of the database in the catalog                                                                                                                                                                                                                                                                                                                                                                            |
| `output.good.table`                    | Required. The name of the table in the database                                                                                                                                                                                                                                                                                                                                                                          |
| `output.good.icebergTableProperties.*` | Optional. A map of key/value strings corresponding to Iceberg's table properties. These can be anything [from the Iceberg table properties documentation](https://iceberg.apache.org/docs/latest/configuration/). The default properties include configuring Iceberg's column-level statistics for the important Snowplow timestamp columns: `load_tstamp`, `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`. |
| `output.good.catalog.options.*`        | Optional. A map of key/value strings which are passed to the catalog configuration. These can be anything [from the Iceberg REST catalog documentation](https://iceberg.apache.org/rest-catalog-spec/), such as authentication credentials.                                                                                                                                                                              |

***

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-lake-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                           |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `input.maxRetries` (since 0.9.0)                    | Optional. Default value 10. Maximum number of retries for AWS SDK operations when reading from Kinesis.                                                                                                                                                                                                                                      |
| `input.apiCallAttemptTimeout` (since 0.10.0)        | Optional. Default value `15 seconds`. Maximum amount of time to wait for an individual Kinesis API call to complete.                                                                                                                                                                                                                         |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |
| `output.bad.maxRetries` (since 0.8.0)               | Optional. Default value 10. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                     |

**GCP:**

| Parameter                                                   | Description                                                                                                                                                                                                                                                                                                                                                             |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                                        | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                          |
| `input.parallelPullFactor`                                  | Optional. Default value 0.5. `parallelPullFactor * cpu count` will determine the number of threads used internally by the Pub/Sub client library for fetching events                                                                                                                                                                                                    |
| `input.durationPerAckExtension`                             | Optional. Default value `600 seconds`. Pub/Sub ack deadlines are extended for this duration when needed. A sensible value is double the size of the `windowing` config parameter, but no higher than 10 minutes.                                                                                                                                                        |
| `input.minRemainingAckDeadline`                             | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `600 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `60 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.maxMessagesPerPull`                                  | Optional. Default value 1000. How many Pub/Sub messages to pull from the server in a single request.                                                                                                                                                                                                                                                                    |
| `input.retries.transientErrors.delay` (since 0.8.0)         | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                              |
| `input.retries.transientErrors.attempts` (since 0.8.0)      | Optional. Default value `10`. Max number of attempts, after which Loader will crash and exit                                                                                                                                                                                                                                                                            |
| `output.bad.topic`                                          | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                     |
| `output.bad.batchSize`                                      | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                       |
| `output.bad.requestByteThreshold`                           | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                          |
| `output.bad.maxRecordSize`                                  | Optional. Default value 10000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                          |
| `output.bad.retries.transientErrors.delay` (since 0.8.0)    | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                              |
| `output.bad.retries.transientErrors.attempts` (since 0.8.0) | Optional. Default value `10`. Max number of attempts, after which Loader will crash and exit                                                                                                                                                                                                                                                                            |

**Azure:**

| Parameter                           | Description                                                                                                                                                                                  |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                   | Required. Name of the Kafka topic for the source of enriched events.                                                                                                                         |
| `input.bootstrapServers`            | Required. Hostname and port of Kafka bootstrap servers hosting the source of enriched events.                                                                                                |
| `input.consumerConf.*`              | Optional. A map of key/value pairs for [any standard Kafka consumer configuration option](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html).      |
| `input.commitTimeout` (since 0.8.0) | Optional. Default value 15 seconds. The time to wait for offset commits to complete. If an offset commit doesn't complete within this time, a CommitTimeoutException will be raised instead. |
| `output.bad.topicName`              | Required. Name of the Kafka topic that will receive failed events.                                                                                                                           |
| `output.bad.bootstrapServers`       | Required. Hostname and port of Kafka bootstrap servers hosting the bad topic                                                                                                                 |
| `output.bad.producerConf.*`         | Optional. A map of key/value pairs for [any standard Kafka producer configuration option](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html).      |
| `output.bad.maxRecordSize`          | Optional. Default value 1000000. Any single failed event sent to Kafka should not exceed this size in bytes                                                                                  |

> **Info:** You can use the `input.consumerConf` and `output.bad.producerConf` options to configure authentication to Azure event hubs using SASL. For example:
>
> ```json
> "input.consumerConf": {
>
>     "security.protocol": "SASL_SSL"
>
>     "sasl.mechanism": "PLAIN"
>
>     "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=<PASSWORD>;"
>
> }
> ```

***

## Other configuration options

| Parameter                                             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `windowing`                                           | Optional. Default value `5 minutes`. Controls how often the loader writes/commits pending events to the lake.                                                                                                                                                                                                                                                                                                                                              |
| `exitOnMissingIgluSchema`                             | Optional. Default value `true`. Whether the loader should crash and exit if it fails to resolve an Iglu Schema. We recommend `true` because Snowplow enriched events have already passed validation, so a missing schema normally indicates an error that needs addressing. Change to `false` so events go the failed events stream instead of crashing the loader.                                                                                        |
| `respectIgluNullability`                              | Optional. Default value `true`. Whether the output parquet files should declare nested fields as non-nullable according to the Iglu schema. When `true`, nested fields are nullable only if they are not required fields according to the Iglu schema. When `false`, all nested fields are defined as nullable in the output table's schemas. Set this to `false` if you use a query engine that dislikes non-nullable nested fields of a nullable struct. |
| `skipSchemas`                                         | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`. A list of schemas that won't be loaded to the lake. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded.                                                                                                                                                         |
| `decompression.maxBytesInBatch` (since 0.10.0)        | Optional. Default value `5242880` (5 MB).The loader automatically detects and decompresses zstd- or gzip-compressed source messages. Uncompressed messages are unaffected.The loader uses this as a cutoff when incrementally adding decompressed events to a batch, and emits the batch as soon as it reaches this size. This protects the loader's memory because a small compressed message can expand into a much larger payload.                      |
| `decompression.maxBytesSinglePayload` (since 0.10.0)  | Optional. Default value `10000000` (10 MB).For zstd- or gzip-compressed source messages, this is the maximum size of a single payload after decompression. The loader emits a [size violation failed event](/docs/api-reference/failed-events/#size-violation) for any payload that exceeds this size.                                                                                                                                                     |
| `spark.conf.*`                                        | Optional. A map of key/value strings which are passed to the internal spark context.                                                                                                                                                                                                                                                                                                                                                                       |
| `spark.taskRetries`                                   | Optional. Default value 3. How many times the internal spark context should be retry a task in case of failure                                                                                                                                                                                                                                                                                                                                             |
| `retries.setupErrors.delay`                           | Optional. Default value `30 seconds`. Configures exponential backoff on errors related to how the lake is set up for this loader. Examples include authentication errors and permissions errors. This class of errors are reported periodically to the monitoring webhook.                                                                                                                                                                                 |
| `retries.transientErrors.delay`                       | Optional. Default value `1 second`. Configures exponential backoff on errors that are likely to be transient. Examples include server errors and network errors.                                                                                                                                                                                                                                                                                           |
| `retries.transientErrors.attempts`                    | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.hostname`                  | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.port`                      | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                                                                                                               |
| `monitoring.metrics.statsd.tags.*`                    | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.period`                    | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.prefix`                    | Optional. Default `snowplow.lakeloader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.metrics.prometheus.tags.*` (since 0.10.0) | Optional. A map of key/value pairs added as common labels on every Prometheus metric. The loader exposes these metrics at the `/metrics` endpoint on the health probe port.                                                                                                                                                                                                                                                                                |
| `monitoring.webhook.endpoint`                         | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how Snowflake is set up for this loader.                                                                                                                                                                                                                                                                            |
| `monitoring.webhook.tags.*`                           | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                                                                                                            |
| `monitoring.webhook.heartbeat.*`                      | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.healthProbe.port`                         | Optional. Default value `8000`. Open a HTTP server that returns OK only if the app is healthy.                                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.healthProbe.unhealthyLatency`             | Optional. Default value `15 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time.                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.dsn`                               | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                                     |
| `monitoring.sentry.tags.*`                            | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                                         |
| `telemetry.disable`                                   | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                                                  |
| `telemetry.userProvidedId`                            | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                                        |
| `inMemBatchBytes`                                     | Optional. Default value 50000000. Controls how many events are buffered in memory before saving the batch to local disk. The default value works well for reasonably sized VMs. For smaller VMs (e.g. less than 2 cpu core, 8 GG memory) consider decreasing this value.                                                                                                                                                                                   |
| `cpuParallelismFraction`                              | Optional. Default value 0.75. Controls how the app splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and cpuParallelismFraction = 0.75, then we process 3 batches concurrently. The default value works well for most workloads.                                                                                                                                                         |
| `numEagerWindows`                                     | Optional. Default value 1. Controls how eagerly the loader starts processing the next timed window even when the previous timed window is still finalizing (committing into the lake). By default, we start processing a timed windows if the previous 1 window is still finalizing, but we do not start processing a timed window if any more older windows are still finalizing. The default value works well for most workloads.                        |
| `http.client.maxConnectionsPerServer`                 | Optional. Default value 4. Configures the internal HTTP client used for Iglu resolver, alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time. For Iglu Server in particular, this avoids overwhelming the server with multiple concurrent requests.                                                                                                                                                          |

---

# Open Table Format Lake Loader
> Load Snowplow events to data lakes using Delta or Iceberg table formats on S3, GCS, or Azure ADLS Gen2.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/

The Lake Loader is an application that loads Snowplow events to a cloud storage bucket using Open Table Formats.

> **Info:** The Lake Loader supports the two major Open Table Formats: [Delta](https://delta.io/) and [Iceberg](https://iceberg.apache.org/).
>
> For Iceberg tables, the loader supports [AWS Glue](https://docs.aws.amazon.com/glue/) and [Iceberg REST](https://iceberg.apache.org/rest-catalog-spec/) as catalogs. The REST catalog integration has been tested with Snowflake Open Catalog.

**AWS:**

On AWS the Lake Loader continually pulls events from Kinesis and writes to S3.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["S3"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any AWS VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-aws:0.11.0
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

snowplow/lake-loader-aws:0.11.0 \

--config=/myconfig/loader.hocon \

--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

**GCP:**

On GCP the Lake Loader continually pulls events from Pub/Sub and writes to GCS.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["GCS"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any GCP VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-gcp:0.11.0
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

snowplow/lake-loader-gcp:0.11.0 \

--config=/myconfig/loader.hocon \

--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

**Azure:**

On Azure the Lake Loader continually pulls events from Kafka and writes to ADLS Gen 2.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["ADLS Gen 2"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any Azure VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-azure:0.11.0
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

snowplow/lake-loader-azure:0.11.0 \

--config=/myconfig/loader.hocon \

--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

***

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.aws.minimal.hocon&#xA;>)

**GCP:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.gcp.minimal.hocon&#xA;>)

**Azure:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.azure.minimal.hocon&#xA;>)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) for all possible configuration parameters.

### Windowing

"Windowing" is an important config setting, which controls how often the Lake Loader commits a batch of events to the data lake. If you adjust this config setting, you should be aware that data lake queries are most efficient when the size of the parquet files in the lake are relatively large.

- If you set this to a **low** value, the loader will write events to the lake more frequently, reducing latency. However, the output parquet files will be smaller, which will make querying the data less efficient.
- Conversely, if you set this to a **high** value, the loader will generate bigger output parquet files, which are efficient for queries — at the cost of events arriving to the lake with more delay.

The default setting is `5 minutes`. For moderate to high volumes, this value strikes a nice balance between the need for large output parquet files and the need for reasonably low latency data.

```text
{

  "windowing": "5 minutes"

}
```

If you tune this setting correctly, then your lake can support efficient analytic queries without the need to run an `OPTIMIZE` job on the files.

### Iglu

The Lake Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

### Metrics

The Lake Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:

| Metric                      | Definition                                                                                                                                                                                                                                      |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_committed`          | A count of events that are successfully written and committed to the lake. Because the loader works in timed windows of several minutes, this metric has a "spiky" value, which is often zero and then periodically spikes up to larger values. |
| `events_received`           | A count of events received by the loader. Unlike `events_committed` this is a smooth varying metric, because the loader is constantly receiving events throughout a timed window.                                                               |
| `events_bad`                | A count of failed events that could not be loaded, and were instead sent to the bad output stream.                                                                                                                                              |
| `latency_millis`            | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader.                                                                                                   |
| `processing_latency_millis` | For each window of events, the time in milliseconds from when the first event is read from the stream, until all events are written and committed to the lake.                                                                                  |
| `e2e_latency_millis`        | The end-to-end latency of the snowplow pipeline. For each window of events, the time in milliseconds from when the first event was received by the collector, until all events are written and committed to the lake.                           |
| `table_data_files_total`    | The total number of data files in the table after a commit. This metric helps monitor table growth and the effectiveness of file compaction strategies.                                                                                         |
| `table_snapshots_retained`  | The number of snapshots retained in the table metadata. This metric helps monitor snapshot accumulation and the effectiveness of snapshot expiration policies.                                                                                  |

See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) for how to configure the StatsD receiver.

**Telemetry notice**

By default, Snowplow collects telemetry data for Lake Loader (since version 0.1.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Lake maintenance jobs
> Maintain Delta Lake and Iceberg tables with OPTIMIZE operations and snapshot expiration to manage file size and storage costs.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/maintenance/

**Delta Lake:**

The [Delta documentation](https://docs.delta.io/latest/best-practices.html#-delta-compact-files) makes recommendations for running regular maintenance jobs to get the best performance from your lake. This guide expands on those recommendations specifically for your Snowplow events lake.

The Snowplow Lake Loader **does not** automatically run the maintenance tasks described below.

## Compact data files

We recommend that you schedule a compaction job to run once per day.

Your daily compaction job should operate on files that were loaded in the previous calendar day, i.e. the last completed timestamp partition. For example, if you run the job via SQL, then use a `WHERE` clause on `load_tstamp_date < current_date()`:

```sql
OPTIMIZE <your_table> WHERE load_tstamp_date < current_date()
```

Data compaction has two benefits for Snowplow data:

1. Queries are more efficient when the underlying parquet files are large. After you compact your files, you will benefit from this whenever you run queries over your historic data, i.e. not just the most recently loaded events.
2. When there are fewer data files, the size of the table's delta log files is smaller. This reduces the overhead of creating a new delta log file, and thus improves the performance of the Lake Loader when committing new events into the lake. This becomes especially important as your lake grows in size over time.

## Vacuum data files

We recommend that you schedule a Vacuum job to run once per week.

A vacuum is needed to clean up unreferenced data files that were logically deleted by the daily compaction jobs.

```sql
VACUUM <your_table>;
```

Unreferenced data files do not negatively impact query performance or write performance. But they do contribute to storage costs.

Vacuum jobs need to list every file in the lake directory. In large lakes, there might be a lot of files, requiring the job to use a lot of Spark compute resources. This is why we recommend to run it infrequently to offset this impact.

**Iceberg:**

The [Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/) makes recommendations for running regular maintenance jobs to get the best performance from your lake. This guide expands on those recommendations specifically for your Snowplow events lake.

The Snowplow Lake Loader **does not** automatically run the maintenance tasks described below.

## Expire snapshots

We recommend that you schedule the `expireSnapshots` Iceberg action to run once per day.

The Snowplow Lake Loader is a continuously-running streaming loader, so it creates new snapshots very frequently, each time it commits more events into the lake. So it is especially important to manage the total number of snapshots held by the Iceberg metadata.

There are two benefits of expiring snapshots in your Snowplow lake:

1. The snapshot metadata files can be much smaller, because the list of metadata files to track is much smaller. This reduces the overhead of creating a new snapshot file, and thus improves the performance of the Lake Loader when committing new events into the lake. This becomes especially important as your lake grows in size over time.
2. If you regularly run compaction jobs (see below) then you will amass lots of small parquet files, which have since been rewritten into larger parquet files. By expiring snapshots, you will delete the redundant small data files, which will save you some storage cost.

For example, if you run the action via a Spark SQL procedure:

```sql
CALL catalog_name.system.expire_snapshots(

    table => 'snowplow.events',

    stream_results => true

)
```

## Compact data files

We recommend that you schedule the `rewriteDataFiles` Iceberg action to run once per day.

Your daily compaction job should operate on files that were loaded in the previous calendar day, i.e. the last completed timestamp partition. For example, if you run the action via a Spark SQL procedure, then use a `where` clause on `load_tstamp < current_date()`:

```sql
CALL catalog_name.system.rewrite_data_files(

    table => 'snowplow.events',

    where => 'load_tstamp < current_date()'

);
```

The `rewriteDataFiles` action has two benefits for Snowplow data:

1. Queries are more efficient when the underlying parquet files are large. After you compact your files, you will benefit from this whenever you run queries over your historic data, i.e. not just the most recently loaded events.
2. When there are fewer data files, the size of the table's manifest files is smaller. This reduces the overhead of creating a new manifest file, and thus improves the performance of the Lake Loader when committing new events into the lake. This becomes especially important as your lake grows in size over time.

## Remove orphan files

We recommend that you schedule the `removeOrphanFiles` Iceberg action to run once per month.

> **Tip:** Your Iceberg table should have `write.metadata.delete-after-commit.enabled=true` set in the table properties. If your Iceberg table was originally created by a Lake Loader older than version 0.7.0, then please run:
>
> ```sql
> ALTER TABLE <your_table>
>
> SET TBLPROPERTIES ('write.metadata.delete-after-commit.enabled'='true')
> ```

As long as `delete-after-commit` is enabled in the table properties, the Snowplow Lake Loader should not create orphan files under normal circumstances. But it is technically still possible for the loader to create orphan files under rare exceptional circumstances, e.g. transient network errors, or if the loader exits without completing a graceful shutdown. Orphan files do not negatively impact query performance or write performance. But they do contribute to storage costs.

This action needs to list every file in the lake directory. In large lakes that might be a very large number of files. In large lakes, there might be a lot of files, requiring the job to use a lot of Spark compute resources. This is why we recommend to run it infrequently to offset this impact.

```sql
CALL catalog_name.system.remove_orphan_files(

    table => 'snowplow.events'

);
```

***

---

# Partitioning for data lakes
> Data lake partitioning by load_tstamp and event_name for efficient querying and incremental processing in Delta and Iceberg tables.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/partitions/

A lake created by the Lake Loader has two levels of partitioning:

1. By the date that the event is loaded to the lake. For Iceberg, we use a [partition transform](https://iceberg.apache.org/spec/#partitioning) `date(load_tstamp)`. For Delta, we create a [generated column](https://delta.io/blog/2023-04-12-delta-lake-generated-columns/) called `date_load_tstamp` defined as `generatedAlwaysAs(CAST(load_tstamp AS DATE))`.
2. By the `event_name` field.

This structure of partitioning works very well with queries that filter on `load_tstamp` and/or `event_name`. It works especially well with incremental models, which only ever process the most recently loaded events.

> **Note:** If you are using Snowplow's DBT packages, then set the `session_timestamp` variable to `load_stamp` to match the table's partitioning.

If you run a query with a clause like `WHERE load_tstamp > ?`, then your query engine can go directly to the partition containing the relevant files. Even better, because Delta and Iceberg collect file-level statistics, such a query can go directly to the relevant files within the partition, matching exactly the `load_tstamp` of interest.

If you often write queries over a single type of event, e.g. `WHERE event_name = 'add_to_cart'` then the query engine can do a very efficient query over the parquet files for the specific event.

> **Note:** The Lake Loader has been optimized for writing into a lake with the default partitioning, and the loader will not perform so well with any other partitioning. For these reasons, we strongly advise that you do not change the partitioning structure of your lake.

---

# S3 loader configuration reference
> Configure S3 Loader with Kinesis input, S3 output, compression, partitioning, and monitoring settings for event archiving.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/

S3 Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run S3 Loader, configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

This is a complete list of the options that can be configured in the S3 loader HOCON config file. The [example configs in github](https://github.com/snowplow/snowplow-s3-loader/tree/master/config) show how to prepare an input file.

| parameter                                                         | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                                | Required. Name of the kinesis stream from which to read                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `input.appName`                                                   | Optional. Default: `snowplow-s3-loader`. Kinesis Client Lib app name (corresponds to DynamoDB table name)                                                                                                                                                                                                                                                                                                                                                                   |
| `input.initialPosition.type` (since 3.0.0)                        | Optional. Default: `TRIM_HORIZON`. Set the initial position to consume the Kinesis stream. Possible values: `LATEST` (most recent data), `TRIM_HORIZON` (oldest available data), `AT_TIMESTAMP` (start from the record at or after the specified timestamp)                                                                                                                                                                                                                 |
| `input.initialPosition.timestamp` (since 3.0.0)                   | Required for `AT_TIMESTAMP`. E.g. `2020-07-17T10:00:00Z`                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `input.retrievalMode.type` (since 3.0.0)                          | Optional. Default: `Polling`. Set the mode for retrieving records. Possible values: `Polling` or `FanOut`                                                                                                                                                                                                                                                                                                                                                                   |
| `input.retrievalMode.maxRecords` (since 3.0.0)                    | Required for `Polling`. Default: `750`. Maximum size of a batch returned by a call to `getRecords`. Records are checkpointed after a batch has been fully processed, thus the smaller `maxRecords`, the more often records can be checkpointed into DynamoDb, but possibly reducing the throughput                                                                                                                                                                          |
| `input.retrievalMode.idleTimeBetweenReads` (since 3.1.0)          | Optional for `Polling`. Default: `1500 millis`. Idle time between `getRecords` requests.                                                                                                                                                                                                                                                                                                                                                                                    |
| `input.workerIdentifier` (since 3.0.0)                            | Optional. Default: host name. Name of this KCL worker used in the DynamoDB lease table                                                                                                                                                                                                                                                                                                                                                                                      |
| `input.leaseDuration` (since 3.0.0)                               | Optional. Default: `10 seconds`. Duration of shard leases. KCL workers must periodically refresh leases in the DynamoDB table before this duration expires                                                                                                                                                                                                                                                                                                                  |
| `input.maxLeasesToStealAtOneTimeFactor` (since 3.0.0)             | Optional. Default: `2.0`. Controls how to pick the max number of leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the KCL to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency                                                                                                                                               |
| `input.checkpointThrottledBackoffPolicy.minBackoff` (since 3.0.0) | Optional. Default: `100 millis`. Minimum backoff before retrying when DynamoDB provisioned throughput exceeded                                                                                                                                                                                                                                                                                                                                                              |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` (since 3.0.0) | Optional. Default: `1 second`. Maximum backoff before retrying when DynamoDB provisioned throughput limit exceeded                                                                                                                                                                                                                                                                                                                                                          |
| `input.debounceCheckpoints` (since 3.0.0)                         | Optional. Default: `10 seconds`. How frequently to checkpoint our progress to the DynamoDB table. By increasing this value, we can decrease the write-throughput requirements of the DynamoDB table                                                                                                                                                                                                                                                                         |
| `input.customEndpoint`                                            | Optional. Override the default endpoint for kinesis client api calls                                                                                                                                                                                                                                                                                                                                                                                                        |
| `input.maxRetries` (since 3.1.0)                                  | Optional. Default: `10`. Maximum number of times the Kinesis client will retry AWS API calls in case of failure                                                                                                                                                                                                                                                                                                                                                             |
| `input.apiCallAttemptTimeout` (since 3.1.0)                       | Optional. Default: `15 seconds`. Maximum time of a single attempt of an AWS API operation.                                                                                                                                                                                                                                                                                                                                                                                  |
| `output.good.path`                                                | Required. Full path to output data, e.g. `s3://acme-snowplow-output/`                                                                                                                                                                                                                                                                                                                                                                                                       |
| `output.good.partitionFormat` (since 2.1.0)                       | Optional. Configures how files are partitioned into S3 directories. When loading self describing jsons, you might choose to partition by `{vendor}.{name}/model={model}/date={yyyy}-{MM}-{dd}`. Valid substitutions are `{vendor}`, `{name}`, `{format}`, `{model}` for self-describing jsons; and `{yyyy}`, `{MM}`, `{dd}`, `{HH}` for year, month, day and hour. Defaults to `{vendor}.{schema}` when loading self-describing JSONs or blank when loading enriched events |
| `output.good.filenamePrefix`                                      | Optional. Add a prefix to files                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `output.good.compression`                                         | Optional. Has to be `GZIP` (default)                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.streamName`                                           | Required. Name of a kinesis stream to output failures                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 3.0.0)      | Optional. Default: `100 milliseconds`. Minimum backoff before retrying when writing fails with exceeded kinesis write throughput                                                                                                                                                                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 3.0.0)      | Optional. Default: `1 second`. Maximum backoff before retrying when writing fails with exceeded kinesis write throughput                                                                                                                                                                                                                                                                                                                                                    |
| `output.bad.recordLimit` (since 3.0.0)                            | Optional. Default: `500`. Maximum allowed to records we are allowed to send to Kinesis in 1 PutRecords request                                                                                                                                                                                                                                                                                                                                                              |
| `output.bad.byteLimit` (since 3.0.0)                              | Optional. Default: `5242880`. Maximum allowed to bytes we are allowed to send to Kinesis in 1 PutRecords request                                                                                                                                                                                                                                                                                                                                                            |
| `output.bad.maxRetries` (since 3.1.0)                             | Optional. Default: `10`. Maximum number of times the Kinesis client will retry AWS API calls in case of failure                                                                                                                                                                                                                                                                                                                                                             |
| `purpose`                                                         | Required. `ENRICHED_EVENTS` for enriched events or `SELF_DESCRIBING` for self-describing data                                                                                                                                                                                                                                                                                                                                                                               |
| `batching.maxBytes` (since 3.0.0)                                 | Optional. Default: `67108864`. After this amount of compressed bytes have been added to the buffer it gets written to a file (unless `maxDelay` is reached before)                                                                                                                                                                                                                                                                                                          |
| `batching.maxDelay` (since 3.0.0)                                 | Optional. Default: `2 minutes`. After this delay has elapsed the buffer gets written to a file (unless `maxBytes` is reached before)                                                                                                                                                                                                                                                                                                                                        |
| `cpuParallelismFactor` (since 3.0.0)                              | Optional. Default: `1`. Controls how the app splits the workload into concurrent batches which can be run in parallel, e.g. if there are 4 available processors and `cpuParallelismFactor = 0.75` then we process 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU                                                                                                                                                   |
| `uploadParallelismFactor` (since 3.0.0)                           | Optional. Default: `2`. Controls number of upload jobs that can be run in parallel, e.g. if there are 4 available processors and `sinkParallelismFraction = 2` then we run 8 upload job concurrently. Adjusting this value can cause the app to use more or less of the available CPU                                                                                                                                                                                       |
| `initialBufferSize` (since 3.0.0)                                 | Optional. Default: none. Overrides the initial size of the byte buffer that holds the compressed events in-memory before they get written to a file. If not set, the initial size is picked dynamically based on other configuration options. The default is known to work well. Increasing this value is a way to reduce in-memory copying, but comes at the cost of increased memory usage                                                                                |
| `decompression.maxBytesInBatch` (since 3.2.0)                     | Optional. Default: `5242880`. The loader automatically detects and decompresses `zstd-` or `gzip-compressed` source messages. Uncompressed messages are unaffected. The loader uses this as a cutoff when incrementally adding decompressed events to a batch, and emits the batch as soon as it reaches this size. This protects the loader's memory because a small compressed message can expand into a much larger payload.                                             |
| `decompression.maxBytesSinglePayload` (since 3.2.0)               | Optional. Default: `10000000`. For `zstd-` or `gzip-compressed` source messages, this is the maximum size of a single payload after decompression. The loader emits a [size violation failed event](/docs/api-reference/failed-events/#size-violation) for any payload that exceeds this size.                                                                                                                                                                              |
| `monitoring.sentry.dsn`                                           | Optional. For tracking uncaught run time exceptions                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `monitoring.metrics.statsd.hostname`                              | Optional. For sending loading metrics (latency and event counts) to a `statsd` server                                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.port`                                  | Optional. Port of the statsd server                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `monitoring.metrics.statsd.tags`                                  | E.g.`{ "key1": "value1", "key2": "value2" }`. Tags are used to annotate the statsd metric with any contextual information                                                                                                                                                                                                                                                                                                                                                   |
| `monitoring.metrics.statsd.prefix`                                | Optional. Default `snowplow.s3loader`. Configures the prefix of statsd metric names                                                                                                                                                                                                                                                                                                                                                                                         |
| `monitoring.metrics.prometheus.tags.*` (since 3.2.0)              | Optional. A map of key/value pairs added as common labels on every Prometheus metric. The loader exposes these metrics at the `/metrics` endpoint on the health probe port.                                                                                                                                                                                                                                                                                                 |
| `monitoring.healthProbe.port` (since 3.0.0)                       | Optional. Default: `8080`. Port of the HTTP server that returns OK only if the app is healthy                                                                                                                                                                                                                                                                                                                                                                               |
| `monitoring.healthProbe.unhealthyLatency` (since 3.0.0)           | Optional. Default: `2 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time                                                                                                                                                                                                                                                                                                                                   |

---

# S3 Loader
> Archive Snowplow events from Kinesis to S3 in LZO or Gzip format for raw payloads, enriched events, and failed events.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/

Snowplow S3 Loader consumes records from an [Amazon Kinesis](http://aws.amazon.com/kinesis/) stream and writes them to [S3](http://aws.amazon.com/s3/). A typical Snowplow pipeline would use the S3 loader in several places:

- Load enriched events from the "enriched" stream. These serve as input for [the RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) when loading to a warehouse.
- Load failed events from the "bad" stream.

Records that can't be successfully written to S3 are written to a second Kinesis stream with the error message.

## Output format : GZIP

The records are treated as byte arrays containing UTF-8 encoded strings (whether CSV, JSON or TSV). New lines are used to separate records written to a file. This format can be used with the Snowplow Kinesis Enriched stream, among other streams.

Gzip encoding is generally used for both enriched data and bad data.

## Running

### Available on Terraform Registry

A Terraform module which deploys the Snowplow S3 Loader on AWS EC2 for use with Kinesis. For installing in other environments, please see the other installation options below.

### Docker image

We publish two different flavours of the docker image:

- `snowplow/snowplow-s3-loader:3.2.0`

- `snowplow/snowplow-s3-loader:3.2.0-distroless` (lightweight alternative)

Here is a standard command to run the loader on a EC2 instance in AWS:

```bash
docker run \

    -d \

    --name snowplow-s3-loader \

    --restart always \

    --log-driver awslogs \

    --log-opt awslogs-group=snowplow-s3-loader \

    --log-opt awslogs-stream='ec2metadata --instance-id' \

    --network host \

    -v $(pwd):/snowplow/config \

    -e 'JAVA_OPTS=-Xms512M -Xmx1024M -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN' \

    snowplow/snowplow-s3-loader:3.2.0 \

    --config /snowplow/config/config.hocon
```

---

# S3 Loader monitoring
> Monitor S3 Loader with StatsD metrics, Sentry error tracking, and Snowplow event tracking for application health and failures.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/monitoring/

The S3 loader has several types of monitoring built in, to help the pipeline operator: Statsd metrics, Sentry alerts, and Snowplow tracking.

## Statsd

[Statsd](https://github.com/statsd/statsd) is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a [pluggable storage backend](https://github.com/statsd/statsd/blob/master/docs/backend.md).

When processing enriched events, the S3 loader can emit metrics to a statsd daemon describing every S3 file it writes. Here is a string representation of the metrics it sends:

```text
snowplow.s3loader.count:42|c|#tag1:value1

snowplow.s3loader.latency_collector_to_load:123|g|#tag1:value1

snowplow.s3loader.latency_millis:56|g|#tag1:value1

snowplow.s3loader.e2e_latency_millis:123|g|#tag1:value1
```

- `count`: total number of events that got written to S3.
- `latency_collector_to_load`: time difference between reaching the collector and getting loaded to S3 (only for enriched events). Will get deprecated eventually in favor of `e2e_latency_millis`.
- `latency_millis`: delay between the input record getting written to the stream and S3 loader starting to process it.
- `e2e_latency_millis`: same as `latency_collector_to_load`, which will get deprecated eventually and replaced with this metric.

Statsd monitoring is configured by setting the `monitoring.metrics.statsd` section in [the hocon file](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/):

```json
"monitoring": {

  "metrics": {

    "hostname": "localhost"

    "port": 8125

    "tags": {

      "tag1": "value1"

      "tag2": "value2"

    }

    "prefix": "snowplow.s3loader"

  }

}
```

## Health probe

Starting with `3.0.0` version S3 loader gets a health probe, configured via the `monitoring.healthProbe` section (see the configuration reference).

## Sentry

[Sentry](https://docs.sentry.io/) is a popular error monitoring service, which helps developers diagnose and fix problems in an application. The S3 loader can send an error report to sentry whenever something unexpected happens. The reasons for the error can then be explored in the sentry server’s UI.

Common reasons might be failure to read or write from Kinesis, or failure to write to S3.

Sentry monitoring is configured by setting the `monitoring.sentry.dsn` key in [the hocon file](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/) with the url of your sentry server:

```json
"monitoring": {

  "dsn": "http://sentry.acme.com"

}
```

---

# S3 Loader 1.0.0 configuration
> Configure S3 Loader 1.0.0 with HOCON settings for Kinesis or NSQ streams, compression, buffering, and monitoring.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/1-0-0-configuration/

The sink is configured using a HOCON file. These are the fields:

- `source`: Choose kinesis or nsq as a source stream
- `sink`: Choose between kinesis or nsq as a sink stream for failed events
- `aws.accessKey` and `aws.secretKey`: Change these to your AWS credentials. You can alternatively leave them as "default", in which case the [DefaultAWSCredentialsProviderChain](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) will be used.
- `kinesis.initialPosition`: Where to start reading from the stream the first time the app is run. "TRIM\_HORIZON" for as far back as possible, "LATEST" for as recent as possibly, "AT\_TIMESTAMP" for after the specified timestamp.
- `kinesis.initialTimestamp`: Timestamp for "AT\_TIMESTAMP" initial position
- `kinesis.maxRecords`: Maximum number of records to read per GetRecords call
- `kinesis.region`: The Kinesis region name to use.
- `kinesis.appName`: Unique identifier for the app which ensures that if it is stopped and restarted, it will restart at the correct location.
- `kinesis.customEndpoint`: Optional endpoint url configuration to override aws kinesis endpoints. This can be used to specify local endpoints when using localstack.
- `kinesis.disableCloudWatch`: Optional override to disable CloudWatch metrics for KCL
- `nsq.channelName`: Channel name for NSQ source stream. If more than one application reading from the same NSQ topic at the same time, all of them must have unique channel name to be able to get all the data from the same topic.
- `nsq.host`: Hostname for NSQ tools
- `nsq.port`: HTTP port number for nsqd
- `nsq.lookupPort`: HTTP port number for nsqlookupd
- `stream.inStreamName`: The name of the input stream of the tool which you choose as a source. This should be the stream to which your are writing records with the Scala Stream Collector.
- `streams.outStreamName`: The name of the output stream of the tool which you choose as sink. This is stream where records are sent if the compression process fails.
- `streams.buffer.byteLimit`: Whenever the total size of the buffered records exceeds this number, they will all be sent to S3.
- `streams.buffer.recordLimit`: Whenever the total number of buffered records exceeds this number, they will all be sent to S3.
- `streams.buffer.timeLimit`: If this length of time passes without the buffer being flushed, the buffer will be flushed. **Note**: With NSQ streams, only record limit is taken into account. Other two option will be ignored.
- `s3.region`: The AWS region for the S3 bucket
- `s3.bucket`: The name of the S3 bucket in which files are to be stored
- `s3.format`: The format the app should write to S3 in (`lzo` or `gzip`)
- `s3.maxTimeout`: The maximum amount of time the app attempts to PUT to S3 before it will kill itself

### Monitoring

It's possible to include Snowplow monitoring in the application. This is setup through the `monitoring` section at the bottom of the config file:

- `monitoring.snowplow.collectorUri` your snowplow collector URI
- `monitoring.snowplow.appId` the app-id used in decorating the events sent

To disable Snowplow monitoring, just remove the entire `monitoring` section from the config.

---

# S3 Loader 2.0.0 upgrade guide
> Upgrade S3 Loader to 2.0.0 with configuration refactoring, purpose property, and new Sentry and StatsD monitoring features.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/2-0-0-upgrade-guide/

## Caution

If you're upgrading from Snowplow pre-R119 and S3 Loader pre-version 0.7.0 you have to upgrade to version 0.7.0 or 1.0.0 first in order to split bad data produced during transition period.

In [Snowplow R119](https://snowplowanalytics.com/blog/2020/05/12/snowplow-release-r119/) we introduced a new self-describing bad rows format. S3 Loader 0.7.0 was the first version capable of partitioning self-describing data based on its schema. 0.7.0 and 1.0.0 are capable to recognize at runtime whether old or new format is consumed and use `partitionedBucket` output path only if necessary, so both formats can be consumed.

S3 Loader 2.0.0 supports only new self-describing format and will be raising exceptions if legacy bad data is pushed.

## Config file

In 2.0.0 the S3 Loader went through a major configuration refactoring. A [sample config](https://github.com/snowplow/snowplow-s3-loader/blob/2.0.0/config/config.hocon.sample) is available in GitHub repository.

- No more `aws` property allowing to hardcode credentials - [default credentials chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) is used
- NSQ support has been dropped
- Instead of `kinesis` and `s3` the topology now is represented as `input` (Kinesis Stream) and `output` (S3 bucket and a Kinesis Stream for bad data)
- `partitionedBucket` property has been removed (see Caution above)
- New `purpose` property allowing Loader to recognize the data it works with: `ENRICHED` for enriched TSVs enabling latency monitoring, `SELF_DESCRIBING` generally for any self-describing JSON but usually used for bad rows and `RAW`

## New features

- `metrics.sentry.dsn` can be used to track exceptions, including internal KCL exceptions
- `metricsd.statsd` can be used to send observability data to StatsD-compatible server

---

# S3 Loader 2.2.x upgrade guide
> Upgrade S3 Loader to 2.2.x with separate Docker images for GZip, LZO, and distroless variants for security improvements.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/2-2-0-upgrade-guide/

Starting from the 2.2.0 release we started publishing three different flavours of the docker image.

- Pull the `:2.2.9` tag if you only need GZip output format

- Pull the `:2.2.9-lzo` tag if you also need LZO output format

- Pull the `:2.2.9-distroless` tag for an lightweight alternative to `:2.2.0`

```bash
docker pull snowplow/snowplow-s3-loader:2.2.9

docker pull snowplow/snowplow-s3-loader:2.2.9-lzo

docker pull snowplow/snowplow-s3-loader:2.2.9-distroless
```

We removed LZO support from the standard image, because it means we can more easily eliminate security vulnerabilities that are brought in from a dependency on hadoop version 2.

The "distroless" docker image is built from [a more lightweight base image](https://github.com/GoogleContainerTools/distroless). It provides some security advantages, because it carries only the minimal files and executables needed for the loader to run.

---

# 3.0.0 upgrade guide
> Guide for upgrading to S3 Loader 3.0.0, covering buffering changes, LZO deprecation, configuration refactoring, and filename changes.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/3-0-0-upgrade-guide/

S3 loader was using AWS SDK v1 which goes EOL at the end of 2025. Bumping to AWS SDK v2 required a full rewrite of the app.

## Buffering

S3 loader buffers the events into memory before writing them to S3. There are 2 key differences between the previous loader and the new one:

- In the previous loader we had one buffer per Kinesis shard, each buffer getting written to one file. In the new loader, records from all the Kinesis shards go to the same buffer and file. The consequence is that the new loader writes fewer but bigger files.

- In the previous loader, records were compressed after the buffer was full, before getting written to disk. In the new loader, records get compressed before getting added to the buffer. The consequence is that the new loader writes bigger files (very close to `maxBytes` if this limit is reached before `maxDelay`).

```mermaid
flowchart LR
  subgraph "Old S3 loader"
    buffer1["buffer"]
    buffer2["buffer"]
    buffer3["buffer"]
  end
  shard1["Kinesis shard"] --> buffer1 -->|"compression"| file1["file/"]
  shard2["Kinesis shard"] --> buffer2 -->|"compression"| file2["file/"]
  shard3["Kinesis shard"] --> buffer3 -->|"compression"| file3["file/"]
  classDef noborder stroke:none,fill:none
  class shard1,shard2,shard3,file1,file2,file3 noborder
```

```mermaid
flowchart LR
  subgraph "New S3 loader"
    buffer["buffer"]
  end
  shard1["Kinesis shard"] & shard2["Kinesis shard"] & shard3["Kinesis shard"] -->|"compression"| buffer --> file["file/"]
  classDef noborder stroke:none,fill:none
  class shard1,shard2,shard3,file noborder
```

## LZO deprecation

Starting from version `3.0.0`, S3 loader should only be used to load enriched events and bad rows (no more `purpose = "RAW"`). The reason for this is that storing the events emitted by the collector is redundant and it is not compatible with features that we have on the roadmap. LZO compression format is not supported any more (it was used in old batch pipelines). Only the following Docker images with GZIP get published:

- `snowplow/snowplow-s3-loader:3.2.0`

- `snowplow/snowplow-s3-loader:3.2.0-distroless` (lightweight alternative)

## Config file

In `3.0.0` S3 Loader went through a major configuration refactoring. A [sample config](https://github.com/snowplow/snowplow-s3-loader/blob/3.0.0/config/config.aws.reference.hocon) is available in GitHub repository.

These config fields have been removed:

- `region`: it is now retrieved from the region provider chain.
- `buffer.recordLimit`: only `maxDelay` and `maxBytes` are now used for the buffering.
- `monitoring.snowplow`: Snowplow tracking (sending events e.g. `app_initialized` or `app_heartbeat`) got removed.
- `output.s3.maxTimeout`

These sections/fields have been renamed:

- `output.s3` -> `output.good`
- `buffer.byteLimit` -> `batching.maxBytes`
- `buffer.timeLimit` -> `batching.maxDelay`
- `input.maxRecords` -> `input.retrievalMode.maxRecords`

For more details, refer to the [configuration reference](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/).

## Change in the filename

There is a change in the name of the files written to S3. In `2.x` the filename was `yyyy-MM-dd-HHmmss-<first_sequence_number_in_shard>-<last_sequence_number_in_shard>.gz`. In `3.0.0` the new filename is `yyyy-MM-dd-HHmmss-<uuid>.gz`. The reason for this change is that S3 loader `2.x` was writing the events one file per shard whereas the new loader is writing the events many shards to the same file.

---

# Upgrade guides for the S3 Loader
> Step-by-step upgrade guides for S3 Loader with configuration changes, breaking changes, and migration paths for major versions.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/

This section contains information to help you upgrade to newer versions of the S3 Loader.

---

# Schema translation to warehouse column types
> How Snowplow JSON schemas map to column types and structures in Redshift, BigQuery, Snowflake, Databricks, Iceberg, and Delta Lake.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/

[Self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number). This page explains what happens to this information in the warehouse.

## Location

Where can you find the data carried by a self-describing event or an entity?

**Redshift:**

Each type of self-describing event and each type of entity get their own dedicated tables. The name of such a table is composed of the schema vendor, schema name and its major version (more on versioning [later](#versioning)).

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting table              |
| --------------------- | ------------------------------------------- | ---------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `com_example_user_1`         |

Inside the table, there will be columns corresponding to the fields in the schema. Their types are determined according to the logic described [below](#types).

> **Note:** The name of each column is the name of the schema field converted to snake case.

> **Note:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the warehouse.

For example, suppose you have the following field in the schema:

```json
"lastName": {

  "type": "string",

  "maxLength": 100

}
```

It will be translated into a column called `last_name` (notice the underscore), of type `VARCHAR(100)`.

**BigQuery:**

**Version 2.x:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

**Version 1.x:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and full schema version (more on versioning [later](#versioning)).

Examples:

| Kind                  | Schema                                      | Resulting column                                       |
| --------------------- | ------------------------------------------- | ------------------------------------------------------ |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1_0_0` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1_0_0`               |

***

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

For self-describing events, the column will be of a `RECORD` type, while for entities the type will be `REPEATED RECORD` (because an event can have more than one entity attached).

Inside the record, there will be fields corresponding to the fields in the schema. Their types are determined according to the logic described [below](#types).

> **Note:** The name of each record field is the name of the schema field converted to snake case.

> **Note:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the warehouse.

For example, suppose you have the following field in the schema:

```json
"lastName": {

  "type": "string",

  "maxLength": 100

}
```

It will be translated into a field called `last_name` (notice the underscore), of type `STRING`.

**Snowflake:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

For self-describing events, the column will be of an `OBJECT` type, while for entities the type will be an `ARRAY` of objects (because an event can have more than one entity attached).

Inside the object, there will be keys corresponding to the fields in the schema. The values for the keys will be of type `VARIANT`.

> **Note:** If an event or entity includes fields not defined in the schema, those fields will be included in the object. However, remember that you need to set `additionalProperties` to `true` in the respective schema for such events and entities to pass schema validation.

For example, suppose you have the following field in the schema:

```json
"lastName": {

  "type": "string",

  "maxLength": 100

}
```

It will be translated into an object with a `lastName` key that points to a value of type `VARIANT`.

**Databricks, Iceberg, Delta:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

For self-describing events, the column will be of a `STRUCT` type, while for entities the type will be `ARRAY` of `STRUCT` (because an event can have more than one entity attached).

Inside the `STRUCT`, there will be fields corresponding to the fields in the schema. Their types are determined according to the logic described [below](#types).

> **Note:** The name of each record field is the name of the schema field converted to snake case.

> **Note:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the warehouse.

For example, suppose you have the following field in the schema:

```json
"lastName": {

  "type": "string",

  "maxLength": 100

}
```

It will be translated into a field called `last_name` (notice the underscore), of type `STRING`.

**Synapse Analytics:**

Each type of self-describing event and each type of entity get their own dedicated columns in the underlying data lake table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

The column will be formatted as JSON — an object for self-describing events and an array of objects for entities (because an event can have more than one entity attached).

Inside the JSON object, there will be fields corresponding to the fields in the schema.

> **Note:** The name of each JSON field is the name of the schema field converted to snake case.

> **Note:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the data lake, and will not be availble in Synapse.

For example, suppose you have the following field in the schema:

```json
"lastName": {

  "type": "string",

  "maxLength": 100

}
```

It will be translated into a field called `last_name` (notice the underscore) inside the JSON object.

***

## Versioning

What happens when you evolve your schema to a [new version](/docs/event-studio/data-structures/versioning/)?

**Redshift:**

Because the table name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new table:

| Schema                                      | Resulting table              |
| ------------------------------------------- | ---------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-0-1` | `com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `com_example_button_press_2` |

When you evolve your schema within the same major version, (non-destructive) changes are applied to the existing table automatically. For example, if you change the `maxLength` of a `string` field, the limit of the `VARCHAR` column would be updated accordingly.

> **Info:** If you make a breaking schema change (e.g. change a type of a field from a `string` to a `number`) without creating a new major schema version, the loader will not be able to modify the table to accommodate the new data.
>
> In this case, _upon receiving the first event with the offending schema_, the loader will instead create a new table, with a name like `com_example_button_press_1_0_1_recovered_9999999`, where:
>
> - `1-0-1` is the version of the offending schema
> - `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one)
>
> To resolve this situation:
>
> - Create a new schema version (e.g. `1-0-2`) that reverts the offending changes and is again compatible with the original table. The data for events with that `1-0-2` schema will start going to the original table as expected.
> - You might also want to manually adapt the data in the `..._recovered_...` table and copy it to the original one.
>
> Note that this behavior was introduced in RDB Loader 6.0.0. In older versions, breaking changes will halt the loading process.

> **Info:** Once the loader creates a column for a given schema version as `NULLABLE` or `NOT NULL`, it will never alter the nullability constraint for that column. For example, if a field is nullable in schema version `1-0-0` and not nullable in version `1-0-1`, the column will remain nullable. (In this example, the Enrich application will still validate data according to the schema, accepting `null` values for `1-0-0` and rejecting them for `1-0-1`.)

**BigQuery:**

**Version 2.x:**

Because the column name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new column:

| Schema                                      | Resulting column                            |
| ------------------------------------------- | ------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-0-1` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2` |

When you evolve your schema within the same major version, (non-destructive) changes are applied to the existing column automatically. For example, if you add a new optional field in the schema, a new optional field will be added to the `RECORD`.

> **Info:** If you make a breaking schema change (e.g. change a type of a field from a `string` to a `number`) without creating a new major schema version, the loader will not be able to modify the column to accommodate the new data.
>
> In this case, _upon receiving the first event with the offending schema_, the loader will instead create a new column, with a name like `unstruct_event_com_example_button_press_1_0_1_recovered_9999999`, where:
>
> - `1-0-1` is the version of the offending schema
> - `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one)
>
> To resolve this situation:
>
> - Create a new schema version (e.g. `1-0-2`) that reverts the offending changes and is again compatible with the original column. The data for events with that schema will start going to the original column as expected.
> - You might also want to manually adapt the data in the `..._recovered_...` column and copy it to the original one.

**Version 1.x:**

Because the column name for the self-describing event or entity includes the full schema version, each version of a schema gets a new column:

| Schema                                      | Resulting column                                |
| ------------------------------------------- | ----------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1_0_0` |
| `com.example/button_press/jsonschema/1-0-1` | `unstruct_event_com_example_button_press_1_0_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2_0_0` |

If you are [modeling your data with dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/), you can use [this macro](https://github.com/snowplow/dbt-snowplow-utils#combine_column_versions-source) to aggregate the data across multiple columns.

> **Info:** While our recommendation is to use major schema versions to indicate breaking changes (e.g. changing a type of a field from a `string` to a `number`), this is not particularly relevant for BigQuery Loader version 1.x. Indeed, each schema version gets its own column, so there is no difference between major and minor versions. That said, we believe sticking to our recommendation is a good idea:
>
> - Breaking changes might affect downstream consumers of the data, even if they don’t affect BigQuery
> - Version 2 of the loader has stricter behavior that matches our loaders for other warehouses and lakes

***

**Snowflake:**

Because the column name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new column:

| Schema                                      | Resulting column                            |
| ------------------------------------------- | ------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-0-1` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2` |

> **Info:** While our recommendation is to use major schema versions to indicate breaking changes (e.g. changing a type of a field from a `string` to a `number`), this is not particularly relevant for Snowflake. Indeed, the event or entity data is stored in the column as is in the `VARIANT` form, so Snowflake is not “aware” of the schema. That said, we believe sticking to our recommendation is a good idea:
>
> - Breaking changes might affect downstream consumers of the data, even if they don’t affect Snowflake
> - In the future, you might decide to migrate to a different data warehouse where our rules are stricter (e.g. Databricks)
>
> Also, creating a new major version of the schema (and hence a new column) is the only way to indicate a change in semantics, where the data is in the same format but has different meaning (e.g. amounts in dollars vs euros).

**Databricks, Iceberg, Delta:**

Because the column name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new column:

| Schema                                      | Resulting column                            |
| ------------------------------------------- | ------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-0-1` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2` |

When you evolve your schema within the same major version, (non-destructive) changes are applied to the existing column automatically. For example, if you add a new optional field in the schema, a new optional field will be added to the `STRUCT`.

> **Info:** If you make a breaking schema change (e.g. change a type of a field from a `string` to a `number`) without creating a new major schema version, the loader will not be able to modify the column to accommodate the new data.
>
> In this case, _upon receiving the first event with the offending schema_, the loader will instead create a new column, with a name like `unstruct_event_com_example_button_press_1_0_1_recovered_9999999`, where:
>
> - `1-0-1` is the version of the offending schema
> - `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one)
>
> To resolve this situation:
>
> - Create a new schema version (e.g. `1-0-2`) that reverts the offending changes and is again compatible with the original column. The data for events with that schema will start going to the original column as expected.
> - You might also want to manually adapt the data in the `..._recovered_...` column and copy it to the original one.

***

## Types

How do schema types translate to the database types?

### Nullability

**Redshift:**

All non-required schema fields translate to nullable columns.

Required fields translate to `NOT NULL` columns:

```json
{

  "properties": {

    "myRequiredField": {"type": ...}

  },

  "required": [ "myRequiredField" ]

}
```

However, it is possible to define a required field where `null` values are allowed (the Enrich application will still validate that the field is present, even if it’s `null`):

```json
"myRequiredField": {

  "type": ["null", ...]

}
```

OR

```json
"myRequiredField": {

  "enum": ["null", ...]

}
```

In this case, the column will be nullable. It does not matter if `"null"` is in the beginning, middle or end of the list of types or enum values.

> **Info:** See also how [versioning](#versioning) affects this.

**BigQuery:**

All non-required schema fields translate to nullable `RECORD` fields.

Required schema fields translate to required `RECORD` fields:

```json
{

  "properties": {

    "myRequiredField": {"type": ...}

  },

  "required": [ "myRequiredField" ]

}
```

However, it is possible to define a required field where `null` values are allowed (the Enrich application will still validate that the field is present, even if it’s `null`):

```json
"myRequiredField": {

  "type": ["null", ...]

}
```

OR

```json
"myRequiredField": {

  "enum": ["null", ...]

}
```

In this case, the `RECORD` field will be nullable. It does not matter if `"null"` is in the beginning, middle or end of the list of types or enum values.

**Snowflake:**

All fields are nullable (because they are stored inside the `VARIANT` type).

**Databricks, Iceberg, Delta:**

All schema fields, including the required ones, translate to nullable fields inside the `STRUCT`.

***

### Types themselves

**Redshift:**

> **Note:** The row order in this table is important. Type lookup stops after the first match is found scanning from top to bottom.

| Json Schema                                                                                                                                                                                                                                                                                                                                                                                          | Redshift Type                                                                                                                                     |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| ```json
{

  "enum": [E1, E2, ...]

}
```The `enum` can contain more than **one** JavaScript type: `string`, `number\|integer`, `boolean`. For the purposes of this `number` and `integer` are the same.`array`, `object`, `NaN` and other types in the `enum` will be cast as fallback `VARCHAR(4096)`._If content size is longer than 4096 it would be truncated when inserted into the Redshift._ | `VARCHAR(M)``M` is the maximum size of `json.stringify(E*)`                                                                                       |
| ```json
{

  "type": ["boolean", "integer"]

}
```OR```json
{

  "type": ["integer", "boolean"]

}
```                                                                                                                                                                                                                                                                                               | `VARCHAR(10)`                                                                                                                                     |
| ```json
{

  "type": [T1, T2, ...]

}
```                                                                                                                                                                                                                                                                                                                                                            | `VARCHAR(4096)`_If content size is longer than 4096 it would be truncated when inserted into the Redshift._                                       |
| ```json
{

  "type": "string",

  "format": "date-time"

}
```                                                                                                                                                                                                                                                                                                                                       | `TIMESTAMP`                                                                                                                                       |
| ```json
{

  "type": "string",

  "format": "date"

}
```                                                                                                                                                                                                                                                                                                                                            | `DATE`                                                                                                                                            |
| ```json
{

  "type": "array"

}
```                                                                                                                                                                                                                                                                                                                                                                  | `VARCHAR(65535)`_Content is stringified and quoted.__If content size is longer than 65535 it would be truncated when inserted into the Redshift._ |
| ```json
{

  "type": "integer",

  "maximum": M

}
```* `M` ≤ 32767                                                                                                                                                                                                                                                                                                                                  | `SMALLINT`                                                                                                                                        |
| ```json
{

  "type": "integer",

  "maximum": M

}
```* 32767 < `M` ≤ 2147483647                                                                                                                                                                                                                                                                                                                     | `INT`                                                                                                                                             |
| ```json
{

  "type": "integer",

  "maximum": M

}
```* `M` >2147483647                                                                                                                                                                                                                                                                                                                              | `BIGINT`                                                                                                                                          |
| ```json
{

  "type": "integer",

  "enum": [E1, E2, ...]

}
```* Maximum `E*` ≤ 32767                                                                                                                                                                                                                                                                                                                | `SMALLINT`                                                                                                                                        |
| ```json
{

  "type": "integer",

  "enum": [E1, E2, ...]

}
```* 32767 < maximum `E*` ≤ 2147483647                                                                                                                                                                                                                                                                                                   | `INT`                                                                                                                                             |
| ```json
{

  "type": "integer",

  "enum": [E1, E2, ...]

}
```* Maximum `E*` > 2147483647                                                                                                                                                                                                                                                                                                           | `BIGINT`                                                                                                                                          |
| ```json
{

  "type": "integer"

}
```                                                                                                                                                                                                                                                                                                                                                                | `BIGINT`                                                                                                                                          |
| ```json
{

  "multipleOf": B

}
```                                                                                                                                                                                                                                                                                                                                                                  | `INT`                                                                                                                                             |
| ```json
{

  "type": "number",

  "multipleOf": B

}
```* Only works for `B`=0.01                                                                                                                                                                                                                                                                                                                    | `DECIMAL(36,2)`                                                                                                                                   |
| ```json
{

  "type": "number"

}
```                                                                                                                                                                                                                                                                                                                                                                 | `DOUBLE`                                                                                                                                          |
| ```json
{

  "type": "boolean"

}
```                                                                                                                                                                                                                                                                                                                                                                | `BOOLEAN`                                                                                                                                         |
| ```json
{

  "type": "string",

  "minLength": M,

  "maxLength": M

}
```* `M` is the same in minLength and maxLength                                                                                                                                                                                                                                                                               | `CHAR(M)`                                                                                                                                         |
| ```json
{

  "type": "string",

  "format": "uuid"

}
```                                                                                                                                                                                                                                                                                                                                            | `CHAR(36)`                                                                                                                                        |
| ```json
{

  "type": "string",

  "format": "ipv6"

}
```                                                                                                                                                                                                                                                                                                                                            | `VARCHAR(39)`                                                                                                                                     |
| ```json
{

  "type": "string",

  "format": "ipv4"

}
```                                                                                                                                                                                                                                                                                                                                            | `VARCHAR(15)`                                                                                                                                     |
| ```json
{

  "type": "string",

  "format": "email"

}
```                                                                                                                                                                                                                                                                                                                                           | `VARCHAR(255)`                                                                                                                                    |
| ```json
{

  "type": "string",

  "maxLength": M

}
```* `enum` is not defined                                                                                                                                                                                                                                                                                                                       | `VARCHAR(M)`                                                                                                                                      |
| ```json
{

  "enum": ["E1"]

}
```* `E1` is the only element                                                                                                                                                                                                                                                                                                                                         | `CHAR(M)``M` is the size of `json.stringify("E1")`                                                                                                |
| If nothing matches above, this is a catch-all.                                                                                                                                                                                                                                                                                                                                                       | `VARCHAR(4096)`_Values will be quoted as in JSON.__If content size is longer than 4096 it would be truncated when inserted into the Redshift._    |

**BigQuery:**

> **Note:** The row order in this table is important. Type lookup stops after the first match is found scanning from top to bottom.

| Json Schema                                                                                                                                                                                                                                                          | BigQuery Type                                    |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| ```json
{

  "type": "object",

  "properties": {...}

}
```If the `"properties"` key is missing, the type for the entire object will be `JSON` instead of `RECORD`.Objects can be nullable. Nested fields can also be nullable (same rules as for everything else). | `RECORD`                                         |
| ```json
{

  "type": "array",

  "items": {...}

}
```The type of the repeated value is determined by the `"items"` key of the schema. If the `"items"` key is missing, then the repeated type is `JSON`.                                                            | `REPEATED`                                       |
| ```json
{

  "type": "string",

  "format": "date-time"

}
```                                                                                                                                                                                                       | `TIMESTAMP`                                      |
| ```json
{

  "type": "string",

  "format": "date"

}
```                                                                                                                                                                                                            | `DATE`                                           |
| ```json
{

  "type": "boolean"

}
```                                                                                                                                                                                                                                | `BOOLEAN`                                        |
| ```json
{

  "type": "string"

}
```                                                                                                                                                                                                                                 | `STRING`                                         |
| ```json
{

  "type": "integer"

}
```                                                                                                                                                                                                                                | `INT`                                            |
| ```json
{

  "type": "number"

}
```OR```json
{

  "type": [ "integer", "number"]

}
```                                                                                                                                                                             | `FLOAT`                                          |
| ```json
{

  "enum": [I1, I2, ...]

}
```* All `Ix` are integer.                                                                                                                                                                                                     | `INT`                                            |
| ```json
{

  "enum": [I1, N1, ...]

}
```* All `Ix`, `Nx` are integer or number.                                                                                                                                                                                     | `FLOAT`                                          |
| ```json
{

  "enum": [S1, S2, ...]

}
```* All `Sx` are strings                                                                                                                                                                                                      | `STRING`                                         |
| ```json
{

  "enum": [A1, A2, ...]

}
```* `Ax` are a mix of different types                                                                                                                                                                                         | `JSON`_String values will be quoted as in JSON._ |
| If nothing matches above, this is a catch-all.                                                                                                                                                                                                                       | `JSON`                                           |

**Snowflake:**

All types are `VARIANT`.

**Databricks, Iceberg, Delta:**

> **Note:** The row order in this table is important. Type lookup stops after the first match is found scanning from top to bottom.

| Json Schema                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | Databricks Type                                                |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| ```json
{

  "type": "object",

  "properties": {...}

}
```The `STRUCT` has nested fields, whose types are determined by the `"properties"` key of the schema.If the `"properties"` key is missing, the type for the entire object will be `STRING` instead of `STRUCT`, and data will be JSON-serialized in the string column.Objects can be nullable. Nested fields can also be nullable (same rules as for everything else).                                                                                                                                                                                                                                                                                                                   | `STRUCT`                                                       |
| ```json
{

  "type": "array",

  "items": {...}

}
```The type of values within the `ARRAY` is determined by the `"items"` key of the schema. If the `"items"` key is missing, then the values within the array will have type `STRING`, and array items will be JSON-serialized.Arrays can be nullable. Nested fields can also be nullable (same rules as for everything else).                                                                                                                                                                                                                                                                                                                                                                   | `REPEATED`                                                     |
| ```json
{

  "type": "string",

  "format": "date-time"

}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `TIMESTAMP`                                                    |
| ```json
{

  "type": "string",

  "format": "date"

}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `DATE`                                                         |
| ```json
{

  "type": "boolean"

}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `BOOLEAN`                                                      |
| ```json
{

  "type": "string"

}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `STRING`                                                       |
| ```json
{

  "type": "integer",

  "minimum": N,

  "maximum": M

}
```* `M` ≤ 2147483647
* `N` ≥ -2147483648                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `INT`                                                          |
| ```json
{

  "type": "integer",

  "minimum": N,

  "maximum": M

}
```* `M` ≤ 9223372036854775807
* `N` ≥ -9223372036854775808                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `BIGINT`                                                       |
| ```json
{

  "type": "integer",

  "minimum": N,

  "maximum": M

}
```* `M` >1e38-1
* `N` <-1e38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `DECIMAL(38,0)`                                                |
| ```json
{

  "type": "integer",

  "minimum": N,

  "maximum": M

}
```* `M` < 1e38-1
* `N` >-1e38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `DOUBLE`                                                       |
| ```json
{

  "type": "integer"

}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `BIGINT`                                                       |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `M` ≤ 2147483647
* `N` ≥ -2147483648
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `INT`                                                          |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `M` ≤ 9223372036854775807
* `N` ≥ -9223372036854775808
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `BIGINT`                                                       |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `M` > 1e38-1
* `N` < -1e38
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DECIMAL(38,0)`                                                |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `M` < 1e38-1
* `N` > -1e38
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DOUBLE`                                                       |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "multipleOf": F

}
```* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `BIGINT`                                                       |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `P` ≤ 38, where `P` is the maximum precision (total number of digits) of `M` and `N`, adjusted for scale (number of digits after the `.`) of `F`.
* `S` is the maximum scale (number of digits after the `.`) in the enum list and it is greater than 0.**More details**`P` = `MAX`(`M.precision` - `M.scale` + `F.scale`, `N.precision` - `N.scale` + `F.scale`)`S` = `F.scale`For example, `M=10.9999, N=-10, F=0.1` will be `DECIMAL(9,1)`. Calculation as follows:`M` is `DECIMAL(6,4)`, `N` is `DECIMAL(2,0)`, `F` is `DECIMAL(2,1)``P` = `MAX`(6 - 4 + 1, 2 + 1) = 3, rounded up to 9`S` = 1result is `DECIMAL(9,1)` | `DECIMAL(P,S)`_`P` is rounded up to either `9`, `18` or `38`._ |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `P` >38, where is the maximum precision (total number of digits) of `M` and `N`, adjusted for scale (number of digits after the `.`) of `F`.
* `S` is the maximum scale (number of digits after the `.`) in the enum list and it is greater than 0.**More details**`P` = `MAX`(`M.precision` - `M.scale` + `F.scale`, `N.precision` - `N.scale` + `F.scale`)For example, `M=10.9999, N=-1e50, F=0.1` will be `DOUBLE`. Calculation as follows:`M` is `DECIMAL(6,4)`, `N` is `DECIMAL(2,0)`, `F` is `DECIMAL(2,1)``P` = `MAX`(6 - 4 + 1, 50 + 1) = 51 >38                                                                   | `DOUBLE`                                                       |
| ```json
{

  "type": "number", // OR ["number", "integer"]

  "minimum": N,

  "maximum": M,

  "multipleOf": F

}
```* `M` < 1e38-1
* `N` > -1e38
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DOUBLE`                                                       |
| ```json
{

  "type": "number" // OR ["number", "integer"]

}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `DOUBLE`                                                       |
| ```json
{

  "enum": [N1, I1, ...]

}
```* All `Nx` and `Ix` are of types number or integer.
* Maximum scale (number of digits after the `.`) in the enum list is 0.
* Maximum absolute value of the enum list is lesser or equal than 2147483647.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `INT`                                                          |
| ```json
{

  "enum": [N1, I1, ...]

}
```* All `Nx` and `Ix` are of types number or integer.
* Maximum scale (number of digits after the `.`) in the enum list is 0.
* Maximum absolute value of the enum list is lesser or equal than 9223372036854775807.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `BIGINT`                                                       |
| ```json
{

  "enum": [N1, I1, ...]

}
```* All `Nx` and `Ix` are of types number or integer.
* Maximum scale (number of digits after the `.`) in the enum list is 0.
* Maximum absolute value of the enum list is greater than 9223372036854775807.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `BIGINT`                                                       |
| ```json
{

  "enum": [N1, I1, ...]

}
```* All `Nx` and `Ix` are of types number or integer.
* Absolute maximum value of the enum list and less than 1e38.
* `S` is the maximum scale (number of digits after the `.`) in the enum list and it is greater than 0.
* `P` is precision (total number of digits in `M`). Rounded up to `9`, `18` or `38`.                                                                                                                                                                                                                                                                                                                                                                                             | `DECIMAL(P,S)`_`P` is rounded up to either `9`, `18` or `38`._ |
| ```json
{

  "enum": [S1, S2, ...]

}
```* All `Sx` are string                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `STRING`                                                       |
| ```json
{

  "enum": [A1, A2, ...]

}
```* `Ax` are a mix of different types                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `STRING`_String values will be quoted as in JSON._             |
| If nothing matches above, this is a catch-all.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `STRING`_Values will be quoted as in JSON._                    |

***

---

# Snowflake Streaming Loader configuration reference
> Configure Snowflake Streaming Loader with Snowpipe Streaming, Kinesis, Pub/Sub, and Kafka settings for real-time warehouse loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/configuration-reference/

The configuration reference in this page is written for Snowflake Streaming Loader `0.6.1`

### License

The Snowflake Streaming Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {

  "accept": true

}
```

### Snowflake configuration

| Parameter                          | Description                                                                                                                                                                                                                                               |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.url`                  | Required, e.g. `https://orgname.accountname.snowflakecomputing.com`. URI of the Snowflake account.                                                                                                                                                        |
| `output.good.user`                 | Required. Snowflake user who has necessary privileges                                                                                                                                                                                                     |
| `output.good.privateKey`           | Required. Snowflake private key, used to connect to the account                                                                                                                                                                                           |
| `output.good.privateKeyPassphrase` | Optional. Passphrase for the private key                                                                                                                                                                                                                  |
| `output.good.role`                 | Optional. Snowflake role which the Snowflake user should assume                                                                                                                                                                                           |
| `output.good.database`             | Required. Name of the Snowflake database containing the events table                                                                                                                                                                                      |
| `output.good.schema`               | Required. Name of the Snowflake schema containing the events table                                                                                                                                                                                        |
| `output.good.table`                | Optional. Default value `events`. Name to use for the events table                                                                                                                                                                                        |
| `output.good.channel`              | Optional. Default value `snowplow`. Prefix to use for the snowflake channels. The full name will be suffixed with a number, e.g. `snowplow-1`. If you run multiple loaders in parallel, then each loader must be configured with a unique channel prefix. |

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-snowflake-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                      |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `input.apiCallAttemptTimeout` (since 0.6.0)         | Optional. Default value `15 seconds`. Maximum amount of time to wait for an individual Kinesis API call to complete.                                                                                                                                                                                                                         |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |

**GCP:**

| Parameter                         | Description                                                                                                                                                                                                                                                                                                                                                           |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`              | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                        |
| `input.durationPerAckExtension`   | Optional. Default value `60 seconds`. Pub/Sub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                               |
| `input.minRemainingAckDeadline`   | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `60 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `6 seconds` left of the remining deadline, before re-extending the message deadline. |
| `output.bad.topic`                | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                   |
| `output.bad.batchSize`            | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                     |
| `output.bad.requestByteThreshold` | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                        |
| `output.bad.maxRecordSize`        | Optional. Default value 9000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                         |

**Azure:**

| Parameter                     | Description                                                                                                                                                                             |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`             | Required. Name of the Kafka topic for the source of enriched events.                                                                                                                    |
| `input.bootstrapServers`      | Required. Hostname and port of Kafka bootstrap servers hosting the source of enriched events.                                                                                           |
| `input.consumerConf.*`        | Optional. A map of key/value pairs for [any standard Kafka consumer configuration option](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html). |
| `output.bad.topicName`        | Required. Name of the Kafka topic that will receive failed events.                                                                                                                      |
| `output.bad.bootstrapServers` | Required. Hostname and port of Kafka bootstrap servers hosting the bad topic                                                                                                            |
| `output.bad.producerConf.*`   | Optional. A map of key/value pairs for [any standard Kafka producer configuration option](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html). |
| `output.bad.maxRecordSize`    | Optional. Default value 1000000. Any single failed event sent to Kafka should not exceed this size in bytes                                                                             |

> **Info:** You can use the `input.consumerConf` and `output.bad.producerConf` options to configure authentication to Azure event hubs using SASL. For example:
>
> ```json
> "input.consumerConf": {
>
>     "security.protocol": "SASL_SSL"
>
>     "sasl.mechanism": "PLAIN"
>
>     "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=<PASSWORD>;"
>
> }
> ```

***

## Other configuration options

| Parameter                                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batching.maxBytes`                                  | Optional. Default value `16000000`. Events are emitted to Snowflake when the batch reaches this size in bytes                                                                                                                                                                                                                                                                                                                         |
| `batching.maxDelay`                                  | Optional. Default value `1 second`. Events are emitted to Snowflake after a maximum of this duration, even if the `maxBytes` size has not been reached                                                                                                                                                                                                                                                                                |
| `batching.uploadParallelismFactor`                   | Optional. Default value 3.5. Controls how many batches can we send simultaneously over the network to Snowflake. E.g. If there are 4 available processors, and `uploadParallelismFactor` is 3.5, then the loader sends up to 14 batches in parallel. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                 |
| `cpuParallelismFactor`                               | Optional. Default value 0.75. Controls how the loaders splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and `cpuParallelismFactor` is 0.75, then the loader processes 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                     |
| `retries.setupErrors.delay`                          | Optional. Default value `30 seconds`. Configures exponential backoff on errors related to how Snowflake is set up for this loader. Examples include authentication errors and permissions errors. This class of errors are reported periodically to the monitoring webhook.                                                                                                                                                           |
| `retries.transientErrors.delay`                      | Optional. Default value `1 second`. Configures exponential backoff on errors that are likely to be transient. Examples include server errors and network errors.                                                                                                                                                                                                                                                                      |
| `retries.transientErrors.attempts`                   | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                                                                                                  |
| `retries.checkCommittedOffset.delay`                 | Optional. Default value `100 millis`. Configures a delay in between flushing events to Snowflake and fetching the latest offset token from Snowflake to check the events are fully ingested.                                                                                                                                                                                                                                          |
| `skipSchemas`                                        | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`. A list of schemas that won't be loaded to Snowflake. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                                                                                      |
| `decompression.maxBytesInBatch` (since 0.6.0)        | Optional. Default value `5242880` (5 MB).The loader automatically detects and decompresses zstd- or gzip-compressed source messages. Uncompressed messages are unaffected.The loader uses this as a cutoff when incrementally adding decompressed events to a batch, and emits the batch as soon as it reaches this size. This protects the loader's memory because a small compressed message can expand into a much larger payload. |
| `decompression.maxBytesSinglePayload` (since 0.6.0)  | Optional. Default value `10000000` (10 MB).For zstd- or gzip-compressed source messages, this is the maximum size of a single payload after decompression. The loader emits a [size violation failed event](/docs/api-reference/failed-events/#size-violation) for any payload that exceeds this size.                                                                                                                                |
| `monitoring.metrics.statsd.hostname`                 | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.metrics.statsd.port`                     | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                                                                                          |
| `monitoring.metrics.statsd.tags.*`                   | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.metrics.statsd.period`                   | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.prefix`                   | Optional. Default `snowplow.snowflake-loader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.prometheus.tags.*` (since 0.6.0) | Optional. A map of key/value pairs added as common labels on every Prometheus metric. The loader exposes these metrics at the `/metrics` endpoint on the health probe port.                                                                                                                                                                                                                                                           |
| `monitoring.webhook.endpoint`                        | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how Snowflake is set up for this loader.                                                                                                                                                                                                                                                       |
| `monitoring.webhook.tags.*`                          | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.webhook.heartbeat.*`                     | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.sentry.dsn`                              | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.sentry.tags.*`                           | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                    |
| `telemetry.disable`                                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                             |
| `telemetry.userProvidedId`                           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                   |
| `output.good.jdbcLoginTimeout`                       | Optional. Sets the login timeout on the JDBC driver which connects to Snowflake                                                                                                                                                                                                                                                                                                                                                       |
| `output.good.jdbcNetworkTimeout`                     | Optional. Sets the network timeout on the JDBC driver which connects to Snowflake                                                                                                                                                                                                                                                                                                                                                     |
| `output.good.jdbcQueryTimeout`                       | Optional. Sets the query timeout on the JDBC driver which connects to Snowflake                                                                                                                                                                                                                                                                                                                                                       |
| `http.client.maxConnectionsPerServer`                | Optional. Default value 4. Configures the internal HTTP client used for alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time.                                                                                                                                                                                                                                                          |

---

# Snowflake Streaming Loader
> Load Snowplow events to Snowflake with sub-minute latency from Kinesis, Pub/Sub, or Kafka using Snowpipe Streaming.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/

The Snowflake Streaming Loader is an application that loads Snowplow events to Snowflake.

> **Tip:** Both [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) and [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) can load data into Snowflake.
>
> Snowflake Streaming Loader is newer and has two advantages:
>
> - Much lower latency — you can get data in Snowflake in seconds, as opposed to minutes with RDB Loader
> - Much lower cost — unlike with RDB Loader, there is no need for EMR and extensive Snowflake compute to load batch files
>
> We recommend the Streaming Loader over the RDB Loader. If you already use RDB Loader, see the [migration guide](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/migrating/) for more information.

**AWS:**

On AWS, the Snowflake Streaming Loader continually pulls events from Kinesis and writes to Snowflake using the [Snowpipe Streaming API](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Snowflake Streaming Loader</b>"}}
subgraph snowflake [Snowflake]
  table[("<b>Events table</b>")]
end
stream-->loader-->|Snowpipe Streaming API|snowflake
```

The Snowflake Streaming Loader is published as a Docker image which you can run on any AWS VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/snowflake-loader-kinesis:0.6.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting the `SNOWFLAKE_PRIVATE_KEY` environment variable so that you can refer to it in the config file.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

--env SNOWFLAKE_PRIVATE_KEY="${SNOWFLAKE_PRIVATE_KEY}" \

snowplow/snowflake-loader-kinesis:0.6.1 \

--config=/myconfig/loader.hocon
```

**GCP:**

On GCP, the Snowflake Streaming Loader continually pulls events from Pub/Sub and writes to Snowflake using the [Snowpipe Streaming API](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Snowflake Streaming Loader</b>"}}
subgraph snowflake [Snowflake]
  table[("<b>Events table</b>")]
end
stream-->loader-->|Snowpipe Streaming API|snowflake
```

The Snowflake Streaming Loader is published as a Docker image which you can run on any GCP VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/snowflake-loader-pubsub:0.6.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting the `SNOWFLAKE_PRIVATE_KEY` environment variable so that you can refer to it in the config file.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

--env SNOWFLAKE_PRIVATE_KEY="${SNOWFLAKE_PRIVATE_KEY}" \

snowplow/snowflake-loader-pubsub:0.6.1 \

--config=/myconfig/loader.hocon
```

**Azure:**

On Azure, the Snowflake Streaming Loader continually pulls events from Kafka and writes to Snowflake using the [Snowpipe Streaming API](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Snowflake Streaming Loader</b>"}}
subgraph snowflake [Snowflake]
  table[("<b>Events table</b>")]
end
stream-->loader-->|Snowpipe Streaming API|snowflake
```

The Snowflake Streaming Loader is published as a Docker image which you can run on any Azure VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/snowflake-loader-kafka:0.6.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting the `SNOWFLAKE_PRIVATE_KEY` environment variable so that you can refer to it in the config file.

```bash
docker run \

--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \

--env SNOWFLAKE_PRIVATE_KEY="${SNOWFLAKE_PRIVATE_KEY}" \

snowplow/snowflake-loader-kafka:0.6.1 \

--config=/myconfig/loader.hocon
```

***

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/main/config/config.kinesis.minimal.hocon&#xA;>)

**GCP:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/main/config/config.pubsub.minimal.hocon&#xA;>)

**Azure:**

```json
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/main/config/config.azure.minimal.hocon&#xA;>)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/configuration-reference/) for all possible configuration parameters.

---

# Migrating to Snowflake Streaming Loader from RDB Loader
> Migrate from RDB Loader to Snowflake Streaming Loader for lower latency and cost with same table or fresh table strategies.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/migrating/

This guide is aimed at Snowplow users who load events into Snowflake via the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/).

We recommend migrating to use the [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) because it has a much lower latency and is cheaper to run.

There are two migration strategies you might take:

1. [Load into the same table as before](#load-into-the-same-table-as-before). This way you have a single events table, containing old events loaded with RDB Loader and new events loaded by the Streaming Loader.
2. [Load into a fresh new table](#load-into-a-fresh-new-table-with-the-streaming-loader) with the Streaming Loader. This is a more cautious approach, but you will need to point any data models or downstream applications, dashboards, etc, to the new table.

## Load into the same table as before

The Streaming Loader is fully compatible with the table created and managed by the recent versions of RDB Loader. In particular, these aspects are exactly the same as before:

- There are 129 columns for the atomic fields, common to all Snowplow events
- [Self-describing events](/docs/fundamentals/events/#self-describing-events) are loaded into columns named like `unstruct_event_com_example_button_press_1`
- [Entities](/docs/fundamentals/entities/) are loaded into columns named like `contexts_com_example_user_1`
- For both self-describing events and entities, a new column is created for each major version of the Iglu schema

> **Tip:** [This page](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) explains how Snowplow data maps to the warehouse in more detail.

You will notice some subtle differences:

#### No loader-side deduplication

RDB Loader performs [within-batch and cross-batch deduplication](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/) during loading. The Streaming Loader does not deduplicate events, so you may see more duplicates than with the RDB Loader.

Snowplow's [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) handle deduplication automatically. If you write custom queries, see [dealing with duplicates](/docs/destinations/warehouses-lakes/querying-data/#dealing-with-duplicates).

#### New `_schema_version` property in entities

Previously, when loading entities into the table, RDB Loader would drop any information about exactly which version of the schema had been used to validate them.

The Streaming Loader adds an extra property called `_schema_version`, so the versioning information is not lost in the warehouse.

For example, if a tracker sends an entity like this:

```json
{

  "schema": "iglu:com.example/my_schema/jsonschema/1-0-3",

  "data": {

    "a": 1

  }

}
```

Then the value loaded into the `contexts_com_example_my_schema_1` column is this:

```json
{

  "a": 1,

  "_schema_version": "1-0-3"

}
```

#### Null values omitted from entities

For self-describing events and entities, the Streaming Loader omits values that are explicitly set to `null`. This differs from RDB loader, which stores them.

For example, if a tracker sends an entity like this:

```json
{

  "schema": "iglu:com.example/my_schema/jsonschema/1-0-0",

  "data": {

    "a": 1,

    "b": null,

    "c": null

  }

}
```

Then the value loaded into the `contexts_com_example_my_schema_1` column is this (note the `b` and `c` fields are missing):

```json
{

  "a": 1

}
```

We made this change as a performance optimization for querying data. The Snowflake docs [explain](https://docs.snowflake.com/en/user-guide/semistructured-considerations) that JSON `null` values affect how Snowflake extracts nested properties. Snowflake automatically builds indexes on the nested properties of VARIANT columns, but only if those properties do not contain explicit `null` values.

#### The `load_tstamp` field is required before migrating

> **Note:** This only affects users migrating from RDB Loader older than version 4.0.0.

The Snowflake events table must have a column named `load_tstamp` of type `TIMESTAMP`. If you have ever used a version of RDB Loader newer than 4.0.0, then it will have already added this column for you. But if you are migrating from an older version of RDB Loader then you will need to add the column manually:

```sql
ALTER TABLE events ADD COLUMN load_tstamp TIMESTAMP
```

## Load into a fresh new table with the Streaming Loader

The Snowflake Streaming Loader will automatically create the events table when you run it for the first time. If you are familiar with the RDB Loader's table, then all of the points in the previous section are still relevant to you. You will also notice a few other differences:

#### No maximum lengths on VARCHAR columns

The old table created by RDB Loader had maximum lengths on some of the columns, e.g. `app_id VARCHAR(255)`. The new Streaming Loader creates columns without max lengths, e.g. `app_id VARCHAR`.

#### VARCHAR instead of CHAR

When RDB Loader created the events table, it used a mixture of VARCHAR and CHAR column types for the various different string fields. For the sake of simplicity, the Streaming Loader uses VARCHAR column types only. In Snowflake, there is no meaningful difference between the two column types.

---

# Postgres Loader for testing and development
> Load Snowplow enriched events into PostgreSQL database from Kinesis or Pub/Sub for development and testing environments.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/

With Snowplow Postgres Loader you can load enriched data or [failed events](/docs/fundamentals/failed-events/) into PostgreSQL database.

> **Danger:** The Postgres loader is not recommended for production use, especially with large data volumes. We recommend using a fully-fledged data warehouse like Databricks, Snowflake, BigQuery or Redshift, together with a [respective loader](/docs/destinations/warehouses-lakes/).

> **Tip:** For more information on how events are stored in Postgres, check the [mapping between Snowplow schemas and the corresponding Postgres column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=postgres).

## Available on Terraform Registry

A Terraform module which deploys the Snowplow Postgres Loader on AWS EC2 for use with Kinesis. For installing in other environments, please see the other installation options below.

## Getting a Docker image

Snowplow Postgres Loader is [published on DockerHub](https://hub.docker.com/r/snowplow/snowplow-postgres-loader):

```bash
docker pull snowplow/snowplow-postgres-loader:0.3.3
```

It accepts very typical configuration for Snowplow Loader:

```bash
docker run --rm \

-v $PWD/config:/snowplow/config \

snowplow/snowplow-postgres-loader:0.3.3 \

--resolver /snowplow/config/resolver.json \

--config /snowplow/config/config.hocon
```

## Iglu

Where `resolver.json` is a typical [Iglu Client](/docs/api-reference/iglu/iglu-resolver/) configuration.

**Please pay attention that schemas for all self-describing JSONs flowing through Postgres Loader must be hosted on Iglu Server 0.6.0 or above.** Iglu Central is static registry and if you use Snowplow-authored schemas - you need to upload all schemas from there as well.

## Configuration

The configuration file is in HOCON format, and it specifies connection details for the target database and the input stream of events.

```json
{

  "input": {

    "type": "Kinesis"

    "streamName": "enriched-events"

    "region": "eu-central-1"

  }

  "output" : {

    "good": {

      "type": "Postgres"

      "host": "localhost"

      "database": "snowplow"

      "username": "postgres"

      "password": ${POSTGRES_PASSWORD}

      "schema": "atomic"

    }

  }

}
```

The `input` section can alternatively specify a GCP PubSub subscription, instead of a kinesis stream like in the example above.

```json
  "input": {

    "type": "PubSub"

    "projectId": "my-project"

    "subscriptionId": "my-subscription"

  }
```

See [the configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/postgres-loader-configuration-reference/) for a complete description of all parameters.

## Other

Loader creates `events` table on the start and every other table when it first encounters its corresponding schema.

You should ensure that the database and schema specified in the configuration exist before starting the loader.

---

# Postgres Loader configuration reference
> Configure Postgres Loader with Kinesis, Pub/Sub, or local input sources and database connection settings for event loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/postgres-loader-configuration-reference/

This is a complete list of the options that can be configured in the postgres loader's HOCON config file. The [example configs in github](https://github.com/snowplow-incubator/snowplow-postgres-loader/tree/master/config) show how to prepare an input file.

|                                  |                                                                                                                                                                                                                                                                                                                   |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.type`                     | Required. Can be "Kinesis", "PubSub" or "Local". Configures where input events will be read from.                                                                                                                                                                                                                 |
| `input.streamName`               | Required when `input.type` is Kinesis. Name of the Kinesis stream to read from.                                                                                                                                                                                                                                   |
| `input.region`                   | Required when `input.type` is Kinesis. AWS region in which the Kinesis stream resides.                                                                                                                                                                                                                            |
| `input.initialPosition`          | Optional. Used when `input.type` is Kinesis. Use "TRIM\_HORIZON" (the default) to start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. Or use "LATEST" to start streaming just after the most recent record in the shard.                                     |
| `input.retrievalMode.type`       | Optional. When `input.type` is Kinesis, this sets the polling mode for retrieving records. Can be "FanOut" (the default) or "Polling".                                                                                                                                                                            |
| `input.retrievalMode.maxRecords` | Optional. Used when `input.retrievalMode.type` is "Polling". Configures how many records are fetched in each poll of the kinesis stream. Default 10000.                                                                                                                                                           |
| `input.projectId`                | Required when `input.type` is PubSub. The name of your GCP project.                                                                                                                                                                                                                                               |
| `input.subscriptionId`           | Required when `input.type` is PubSub. Id of the PubSub subscription to read events from                                                                                                                                                                                                                           |
| `input.path`                     | Required when `input.type` is Local. Path for event source. It can be directory or file. If it is directory, all the files under given directory will be read recursively. Also, given path can be both absolute path or relative path w\.r.t. executable.                                                        |
| `output.good.host`               | Required. Hostname of the postgres database.                                                                                                                                                                                                                                                                      |
| `output.good.port`               | Optional. Port number of the postgres database. Default 5432.                                                                                                                                                                                                                                                     |
| `output.good.database`           | Required. Name of the postgres database.                                                                                                                                                                                                                                                                          |
| `output.good.username`           | Required. Postgres role name to use when connecting to the database                                                                                                                                                                                                                                               |
| `output.good.password`           | Required. Password for the postgres user.                                                                                                                                                                                                                                                                         |
| `output.good.schema`             | Required. The Postgres schema in which to create tables and write events.                                                                                                                                                                                                                                         |
| `output.good.sslMode`            | Optional. Configures how the client and server agree on ssl protection. Default "REQUIRE"                                                                                                                                                                                                                         |
| `output.bad.type`                | Optional. Can be "Kinesis", "PubSub", "Local" or "Noop". Configures where failed events will be sent. Default is "Noop" which means failed events will be discarded                                                                                                                                               |
| `output.bad.streamName`          | Required when `bad.type` is Kinesis. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                      |
| `output.bad.region`              | Required when `bad.type` is Kinesis. AWS region in which the Kinesis stream resides.                                                                                                                                                                                                                              |
| `output.bad.projectId`           | Required when `bad.type` is PubSub. The name of your GCP project.                                                                                                                                                                                                                                                 |
| `output.bad.topicId`             | Required when `bad.type` is PubSub. Id of the PubSub topic to write failed events to                                                                                                                                                                                                                              |
| `output.bad.path`                | Required when `bad.type` is Local. Path of the file to write failed events                                                                                                                                                                                                                                        |
| `purpose`                        | Optional. Set this to "ENRICHED\_EVENTS" (the default) when reading the stream of enriched events in tsv format. Set this to "JSON" when reading a stream of self-describing json, e.g. snowplow [bad rows](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows). |
| `monitoring.metrics.cloudWatch`  | Optional boolean, with default true. For kinesis input, this is used to disable sending metrics to cloudwatch.                                                                                                                                                                                                    |

#### Advanced options

We believe these advanced options are set to sensible defaults, and hopefully you won't need to ever change them.

|                                          |                                                                                                                                                                                                           |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backoffPolicy.minBackoff`               | If producer (PubSub or Kinesis) fails to send item, it will retry to send it again. This field configures backoff time for first retry. Every retry will double the backoff time of previous one.         |
| `backoffPolicy.maxBackoff`               | Maximum backoff time for retry. After this value is reached, backoff time will no more increase.                                                                                                          |
| `input.checkpointSettings.maxBatchSize`  | Used when `input.type` is Kinesis. Determines the max number of records to aggregate before checkpointing the records. Default is 1000.                                                                   |
| `input.checkpointSettings.maxBatchWait`  | Used when `input.type` is Kinesis. Determines the max amount of time to wait before checkpointing the records. Default is 10 seconds.                                                                     |
| `input.checkpointSettings.maxConcurrent` | Used when `input.type` is PubSub. The max number of concurrent evaluation for checkpointer.                                                                                                               |
| `output.good.maxConnections`             | Maximum number of connections database pool is allowed to reach. Default 10                                                                                                                               |
| `output.good.threadPoolSize`             | Size of the thread pool for blocking database operations. Default is value of "maxConnections"                                                                                                            |
| `output.bad.delayThreshold`              | Set the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. Default 200 milliseconds |
| `output.bad.maxBatchSize`                | A batch of messages will be emitted when the number of events in batch reaches the given size. Default 500                                                                                                |
| `output.bad.maxBatchBytes`               | A batch of messages will be emitted when the size of the batch reaches the given size. Default 5 MB                                                                                                       |

---

# RDB Loader for Redshift and Databricks
> Load Snowplow events into Redshift, Databricks, or Snowflake with transformation and deduplication using Spark or stream transformers.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/

We use the name RDB Loader (from "relational database") for a set of applications that can be used to load Snowplow events into a data warehouse. Use these tools if you want to load into **Redshift** (including Redshift serverless), **Databricks**, or **Snowflake** (the latter not recommended). For other destinations, see [here](/docs/api-reference/loaders-storage-targets/).

**Redshift:**

**AWS (Batching, recommended):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Redshift, stores them in an S3 bucket and instructs Redshift to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

  s3loader{{"<b>S3 Loader</b>"}}
  prebucket[("<b>Enriched Events</b>
(S3 bucket)")]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Redshift"
  table[("<b>Events table</b>")]
  
end
stream-->s3loader-->prebucket-->loader-->bucket--->Redshift
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Redshift.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to TSV
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Redshift: Send SQL commands for loading
  Note over Redshift: Load the data from the S3 bucket using “COPY FROM”
end
```

**AWS (Micro-batching):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Redshift, stores them in an S3 bucket and instructs Redshift to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Redshift"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Redshift
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Redshift.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to TSV
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Redshift: Send SQL commands for loading
  Note over Redshift: Load the data from the S3 bucket using “COPY FROM”
end
```

***

**Databricks:**

> **Note:** The cloud selection below is for your _pipeline_. We don’t have restrictions on where Databricks itself is deployed.

**AWS (Batching, recommended):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Databricks, stores them in an S3 bucket and instructs Databricks to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

  s3loader{{"<b>S3 Loader</b>"}}
  prebucket[("<b>Enriched Events</b>
(S3 bucket)")]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Databricks"
  table[("<b>Events table</b>")]
  
end
stream-->s3loader-->prebucket-->loader-->bucket--->Databricks
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Databricks.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to Parquet
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Databricks: Send SQL commands for loading
  Note over Databricks: Load the data from the S3 bucket using “COPY FROM”
end
```

**AWS (Micro-batching):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Databricks, stores them in an S3 bucket and instructs Databricks to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Databricks"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Databricks
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Databricks.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to Parquet
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Databricks: Send SQL commands for loading
  Note over Databricks: Load the data from the S3 bucket using “COPY FROM”
end
```

**GCP:**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Databricks, stores them in an GCS bucket and instructs Databricks to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(GCS bucket)")]
subgraph "Databricks"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Databricks
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Databricks.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to Parquet
  
  Note over Transformer: Write data to the GCS bucket
  Transformer->>Loader: Notify the loader (via Pub/Sub)
  Loader->>Databricks: Send SQL commands for loading
  Note over Databricks: Load the data from the GCS bucket using “COPY FROM”
end
```

***

**Snowflake:**

> **Tip:** Both [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) and [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) can load data into Snowflake.
>
> Snowflake Streaming Loader is newer and has two advantages:
>
> - Much lower latency — you can get data in Snowflake in seconds, as opposed to minutes with RDB Loader
> - Much lower cost — unlike with RDB Loader, there is no need for EMR and extensive Snowflake compute to load batch files
>
> We recommend the Streaming Loader over the RDB Loader. If you already use RDB Loader, see the [migration guide](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/migrating/) for more information.

> **Note:** The cloud selection below is for your _pipeline_. We don’t have restrictions on where Snowflake itself is deployed.

**AWS (Batching, recommended):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an S3 bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

  s3loader{{"<b>S3 Loader</b>"}}
  prebucket[("<b>Enriched Events</b>
(S3 bucket)")]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->s3loader-->prebucket-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the S3 bucket using “COPY FROM”
end
```

**AWS (Micro-batching):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an S3 bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the S3 bucket using “COPY FROM”
end
```

**GCP:**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an GCS bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(GCS bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the GCS bucket
  Transformer->>Loader: Notify the loader (via Pub/Sub)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the GCS bucket using “COPY FROM”
end
```

**Azure:**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an Azure Blob Storage bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(Azure Blob Storage bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the Azure Blob Storage bucket
  Transformer->>Loader: Notify the loader (via Kafka)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the Azure Blob Storage bucket using “COPY FROM”
end
```

***

***

> **Tip:** For more information on how events are stored in the warehouse, check the [mapping between Snowplow schemas and the corresponding warehouse column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

To run RDB Loader, you will need to run one instance of the Transformer and one instance of the Loader.

## How to pick a transformer

The transformer app currently comes in two flavours: a Spark job that processes data in batches, and a long-running streaming app.

The process of transforming the data is not dependent on the storage target. Which one is best for your use case depends on three factors:

- cloud provider you want to use (AWS, GCP or Azure)
- your expected data volume
- how much importance you place on deduplicating the data before loading it into the data warehouse.

### Based on cloud provider

If you want to run the transformer on AWS, you can use Spark transformer (`snowplow-transformer-batch`) or Transformer Kinesis (`snowplow-transformer-kinesis`).

If you want to run the transformer on GCP, you can use Transformer Pubsub (`snowplow-transformer-pubsub`).

If you want to run the transformer on Azure, you can use Transformer Kafka (`snowplow-transformer-kafka`).

### Based on expected data volume

The Spark transformer (`snowplow-transformer-batch`) is the best choice for big volumes, as the work can be split across multiple workers. However, the need to run it on EMR creates some overhead that is not justified for low-volume pipelines.

The stream transformer (`snowplow-transformer-kinesis`, `snowplow-transformer-pubsub` and `snowplow-transformer-kafka`) is a much leaner alternative and suggested for use with low volumes that can be comfortably processed on a single node. However, multiple stream transformers can be run parallel therefore it is possible to process big data volume with stream transformer too.

To make the best choice, consider:

- What is the underlying infrastructure? For example, a single-node stream transformer will perform differently based on the resources it is given by the machine it runs on.
- What is the frequency for processing data? For example, even in a low-volume pipeline, if you only run the transform job once a day, the accumulated data might be enough to justify the use of Spark.

### Based on the importance of deduplication

The transformer is also in charge of [deduplicating](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/) the data. Currently, only the Spark transformer can do that.

If duplicates are not a concern, or if you are happy to deal with them after the data has been loaded in the warehouse, then pick a transformer based on your expected volume (see above). Otherwise, use the Spark transformer.

## How to pick a loader based on the destination

There are different loader applications depending on the storage target. Currently, RDB Loader supports Redshift (AWS only), Snowflake and Databricks.

For loading into **Redshift** (including Redshift serverless), use the `snowplow-rdb-loader-redshift` artifact.

For loading into **Snowflake**, use the `snowplow-rdb-loader-snowflake` artifact.

For loading into **Databricks**, use the `snowplow-rdb-loader-databricks` artifact.

## How `transformer` and `loader` interface with other Snowplow components and each other

The applications communicate through messages.

The transformer consumes enriched tsv-formatted Snowplow events from S3 (AWS) or stream (AWS, GCP and Azure). It writes its output to blob storage (S3, GCS or Azure Blob Storage). Once it's finished processing a batch of data, it issues a message with details about the run.

The loader consumes a stream of these messages and uses them to determine what data needs to be loaded. It issues the necessary SQL commands to the storage target.

---

# Load into Databricks using the RDB Loader
> Load wide row Parquet data into Databricks with automatic schema creation and Delta Lake optimization.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/databricks-loader/

To set up the Databricks loader, the following resources need to be created:

- [Databricks cluster](https://docs.databricks.com/clusters/create-cluster.html)
- [Databricks access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html)

The `events` table and the database schema will be created automatically by the loader. You can [configure](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/) the name of the database schema with the `storage.schema` config field. The table name (`events`) can’t be changed.

Keep in mind that the Databricks Loader database user needs to have permissions to create schemas on the given database to be able to perform this operation. Check [this page](https://docs.databricks.com/sql/language-manual/security-grant.html) for more information about granting privileges in Databricks. You can also create the schema manually if you prefer.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/rdb-loader-databricks:6.6.0`.

## Configuring `rdb-loader-databricks`

The loader takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Databricks loader can be found [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by RDB Loader **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) version 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Databricks loader

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/rdb-loader-databricks:6.6.0 \

--iglu-config $RESOLVER_BASE64 \

--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Databricks Loader (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Loading transformed data into warehouses
> Load transformed Snowplow data into Redshift, Snowflake, or Databricks with automated table management and SQL COPY operations.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/

_For a high-level overview of the RDB Loader architecture, of which the loader is a part, see [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)._

The loader applications are specialised to a specific storage target. Each one performs 3 key tasks:

- Consume messages from SQS / SNS / Pubsub / Kafka to discover information about transformed data: where it is stored and what it looks like.
- Use the information from the message to determine if any changes to the target table(s) are required, eg to add a column for a new event field. If required, submit the appropriate SQL statement for execution by the storage target.
- Prepare and submit for execution the appropriate SQL `COPY` statement.

For loading into **Redshift**, use the [Redshift loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/redshift-loader/). This loads [shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) into multiple Redshift tables.

For loading into **Snowflake**, use the [Snowflake loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/snowflake-loader/). This loads [wide row JSON format data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format) into a single Snowflake table.

For loading into **Databricks**, use the [Databricks loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/databricks-loader/). This loads [wide row Parquet format data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format) into a single Databricks table.

> **Note:** AWS is fully supported for both Snowflake and Databricks. GCP is supported for Snowflake (since 5.0.0). Azure is supported for Snowflake (since 5.7.0).

---

# Monitoring RDB Loader
> Monitor RDB Loader with folder checks, health checks, StatsD metrics, Sentry alerts, and Snowplow tracking for warehouse loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/monitoring/

The loader app has several types of monitoring built in to help the pipeline operator: folder monitoring, warehouse health checks, StatsD metrics, Sentry alerts, and Snowplow tracking.

For all monitoring configuration options, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

## Webhook alerts

The loader can send `POST` requests via HTTP webhook to a configurable URL whenever there is an issue which needs investigation by the pipeline operator. The webhook payload conforms to the [`alert`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.monitoring.batch/alert/jsonschema/1-0-0) schema on Iglu Central.

You can configure where the webhook is sent by setting the `monitoring.webhook` section in the `config.hocon` file.

The webhook monitoring can be used for folder monitoring and warehouse health checks.

### Folder monitoring

A webhook alert is sent whenever the loader identifies inconsistencies between the transformed output in S3 and the data in the warehouse. The algorithm is as follows:

- Check if all folders on S3 have a `shredding_complete.json` file (the legacy name is kept for backwards compatibility, but this applies to wide row format data as well). A missing file suggests the transformer failed to complete writing the transformed data, and so manual intervention is required to remove the folder from S3 and rerun.
- Check if all folders on S3 created within a specific time range are listed in the warehouse manifest table. This table is maintained by the loader and contains information about loads. If a folder is missing from the manifest table, it suggests the loader has previously tried and failed to load it. Manual intervention is required to resend the `shredding_complete.json` message via SQS / SNS to trigger reloading of the folder.

Folder monitoring is configured by setting the `monitoring.folders` section in the `config.hocon` file.

### Warehouse health check

The loader can send an alert if the warehouse does not respond to a periodic `SELECT 1` statement. For each failed health check, a `POST` request is sent via the webhook.

The health check is configured by setting the `monitoring.healthCheck` section in the `config.hocon` file.

## StatsD and stdout

[StatsD](https://github.com/statsd/statsd) is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a [pluggable storage backend](https://github.com/statsd/statsd/blob/master/docs/backend.md).

The loader can emit metrics to a StatsD daemon describing every batch it processes. Here is a string representation of the metrics it sends:

```text
snowplow.rdbloader.count_good:42|c|#tag1:value1

snowplow.rdbloader.count_bad:2|c|#tag1:value1

snowplow.rdbloader.latency_collector_to_load_min:123.4|g|#tag1:value1

snowplow.rdbloader.latency_collector_to_load_max:234.5|g|#tag1:value1

snowplow.rdbloader.latency_transformer_start_to_load:66.6|g|#tag1:value1

snowplow.rdbloader.latency_transformer_end_to_load:44.4|g|#tag1:value1
```

These are the meanings of the individual metrics:

- `count_good`: the total number of good events in the batch that was loaded
- `count_bad`: the total number of bad events in the batch that was loaded (available since version 5.4.0)
- `latency_collector_to_load_min`: for the most recent event in the batch, this is the time difference between reaching the collector and getting loaded to the warehouse
- `latency_collector_to_load_min`: for the oldest event in the batch, this is the time difference between reaching the collector and getting loaded to the warehouse
- `latency_transformer_start_to_load`: time difference between the transformer starting on this batch and the loader completing loading to the warehouse
- `latency_transformer_end_to_load`: time difference between the transformer completing this batch and the loader completing loading it into the warehouse.

StatsD monitoring is configured by setting the `monitoring.metrics.statsd` section in the `config.hocon` file.

You can expose these metrics in `stdout` for easier debugging by setting the `monitoring.metrics.stdout` section in the `config.hocon` file.

## Sentry

[Sentry](https://docs.sentry.io/) is a popular error monitoring service, which helps developers diagnose and fix problems in an application. The loader and transformer can both send an error report to sentry whenever something unexpected happens. The reasons for the error can then be explored in the Sentry server’s UI.

Common reasons might be lost connection to the database, or an HTTP error fetching a schema from an Iglu server.

Sentry monitoring is configured by setting the `monitoring.sentry.dsn` section in the `config.hocon` file.

## Snowplow tracking

The loader can emit a Snowplow event to a collector when the application crashes with an unexpected error. The event conforms to the [`load_failed`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.monitoring.batch/load_failed/jsonschema/1-0-0) schema on Iglu Central.

Snowplow tracking is configured by setting the `monitoring.snowplow` section in the `config.hocon` file.

---

# Load into Redshift using the RDB Loader
> Load shredded Snowplow events into Amazon Redshift with automatic schema creation and table management.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/redshift-loader/

The `events` table and the database schema will be created automatically by the loader. You can [configure](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/) the name of the database schema with the `storage.schema` config field. The table name (`events`) can’t be changed.

Keep in mind that the Redshift Loader database user needs to have permissions to create schemas on the given database to be able to perform this operation. Check [this page](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) for more information about granting privileges in Redshift. You can also create the schema manually if you prefer.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/rdb-loader-redshift:6.6.0`.

## Configuring `rdb-loader-redshift`

The loader takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Redshift loader can be found [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by RDB Loader **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Redshift loader

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/rdb-loader-redshift:6.6.0 \

--iglu-config $RESOLVER_BASE64 \

--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Redshift Loader (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Load into Snowflake using the RDB Loader
> Load wide row JSON data into Snowflake on AWS, GCP, or Azure with TempCreds or NoCreds authentication methods.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/snowflake-loader/

It is possible to run Snowflake Loader on AWS, GCP and Azure.

### Setting up Snowflake

You can use the steps outlined in our [quick start guide](/docs/get-started/self-hosted/quick-start/?warehouse=snowflake#prepare-the-destination) to create most of the necessary Snowflake resources.

There are two different authentication methods with Snowflake Loader:

- With the `TempCreds` method, there are no additional Snowflake resources needed.
- With the `NoCreds` method, the Loader needs a Snowflake stage.

This choice is controlled by the `loadAuthMethod` [configuration setting](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/#snowflake-loader-storage-section).

> **Note:** For GCP pipelines, only the `NoCreds` method is available.

**Using the NoCreds method**

First, create a Snowflake stage. For that, you will need a Snowflake database, Snowflake schema, Snowflake storage integration, Snowflake file format, and the path to the transformed events bucket (in S3, GCS or Azure Blob Storage).

You can follow [this tutorial](https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html) to create the storage integration.

Assuming you created the other required resources for it, you can create the Snowflake stage by following [this document](https://docs.snowflake.com/en/sql-reference/sql/create-stage.html).

Finally, use the `transformedStage` [configuration setting](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/#snowflake-loader-storage-section) to point the loader to your stage.

### Running the loader

There are dedicated terraform modules for deploying Snowflake Loader on [AWS](https://registry.terraform.io/modules/snowplow-devops/snowflake-loader-ec2/aws/latest) and [Azure](https://github.com/snowplow-devops/terraform-azurerm-snowflake-loader-vmss). You can see how they are used in our full pipeline deployment examples [here](/docs/get-started/self-hosted/quick-start/).

We don't have a terraform module for deploying Snowflake Loader on GCP yet. Therefore, it needs to be deployed manually at the moment.

### Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/rdb-loader-snowflake:6.6.0`.

### Configuring `rdb-loader-snowflake`

The loader takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

| Minimal Configuration                                                                                                                                  | Extended Configuration                                                                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [aws/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.minimal.hocon)     | [aws/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.reference.hocon)     |
| [gcp/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.minimal.hocon)     | [gcp/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.reference.hocon)     |
| [azure/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.minimal.hocon) | [azure/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.reference.hocon) |

For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by RDB Loader **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

### Running the Snowflake loader

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/rdb-loader-snowflake:6.6.0 \

--iglu-config $RESOLVER_BASE64 \

--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Snowflake Loader (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# RDB Loader configuration reference
> Configure RDB Loader for Redshift, Snowflake, and Databricks with storage, messaging, scheduling, and monitoring settings.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/

The configuration reference in this page is written for RDB Loader 5.0.0 or higher.

The configuration reference pages for previous versions can be found [here](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/rdb-loader-previous-versions/).

| Minimal Configuration                                                                                                                                  | Extended Configuration                                                                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [aws/redshift.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.minimal.hocon)       | [aws/redshift.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.reference.hocon)       |
| [aws/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.minimal.hocon)     | [aws/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.reference.hocon)     |
| [aws/databricks.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.minimal.hocon)   | [aws/databricks.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.reference.hocon)   |
| [gcp/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.minimal.hocon)     | [gcp/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.reference.hocon)     |
| [azure/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.minimal.hocon) | [azure/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.reference.hocon) |

All applications use a common module for core functionality, so only the `storage` sections are different in their config.

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

## Password rotation

To rotate the password for your RDB Loader, you'll need to contact [Snowplow Support](https://support.snowplow.io/), and share the intended password using the secure Credential sharing form in Console. To avoid disruption to your pipeline, we'll coordinate a time with you to update the credentials.

## Redshift Loader `storage` section

| Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`                                        | Optional. The only valid value is the default: `redshift`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `host`                                        | Required. Host name of Redshift cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `port`                                        | Required. Port of Redshift cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `database`                                    | Required. Redshift database which the data will be loaded to.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `roleArn`                                     | Required if 'NoCreds' is chosen as load auth method.. AWS Role ARN allowing Redshift to load data from S3.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `schema`                                      | Required. Redshift schema name, eg “atomic”.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `username`                                    | Required. DB user with permissions to load data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `jdbcAuth.*` (since 6.3.0)                    | Required. JDBC authentication configuration. Supports two modes: `password` and `iam`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `jdbcAuth.type` (since 6.3.0)                 | Required. The authentication type. Possible values: `password` and `iam`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `jdbcAuth.password` (since 6.3.0)             | Required if `jdbcAuth.type` is `password`. Password of the DB user.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `jdbcAuth.roleArn` (since 6.3.0)              | Required if `jdbcAuth.type` is `iam`. IAM role ARN with permissions to call `GetClusterCredentials`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `jdbcAuth.roleExternalId` (since 6.3.0)       | Required if `jdbcAuth.type` is `iam`. External ID for assuming the IAM role, used to restrict role assumption to trusted principals.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `jdbcAuth.redshiftRegion` (since 6.3.0)       | Required if `jdbcAuth.type` is `iam`. AWS region where the Redshift cluster is located.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `jdbcAuth.clusterId` (since 6.3.0)            | Required if `jdbcAuth.type` is `iam`. Redshift cluster identifier.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `jdbcAuth.roleSessionName` (since 6.3.0)      | Optional. Session name for the assumed IAM role.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `jdbcAuth.credentialsTtl` (since 6.3.0)       | Optional. TTL for temporary database credentials when using IAM authentication. Must be between 15 minutes and one hour.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `maxError`                                    | Optional. Configures the [Redshift MAXERROR load option](https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-data-load.html#copy-maxerror). The default is 10.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `loadAuthMethod.*` (since 5.2.0)              | Optional, default method is `NoCreds`. Specifies the auth method to use with the `COPY` statement.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `loadAuthMethod.type`                         | Required if `loadAuthMethod` section is included. Specifies the type of the authentication method. The possible values are `NoCreds` and `TempCreds`. With `NoCreds`, no credentials will be passed to the `COPY` statement. Instead, Redshift cluster needs to be configured with an AWS Role ARN that allows it to load data from S3. This Role ARN needs to be passed in the `roleArn` setting above. You can find more information [here](https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html). With 'TempCreds', temporary credentials will be created for every load operation and these temporary credentials will be passed to the `COPY` statement. |
| `loadAuthMethod.roleArn`                      | Required if `loadAuthMethod.type` is `TempCreds`. IAM role that is used while creating temporary credentials. This role should allow access to the S3 bucket the transformer will write data to, with the following permissions: `s3:GetObject*`, `s3:ListBucket`, and `s3:GetBucketLocation`.                                                                                                                                                                                                                                                                                                                                                                                                   |
| `loadAuthMethod.credentialsTtl` since (5.4.0) | Optional, default value `1 hour`. If `TempCreds` load auth method is used, this value will be used as a session duration of temporary credentials used for loading data and folder monitoring. In that case, it can't be greater than 1 hour and can't be less than 15 minutes.                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.*`                                      | Optional. Custom JDBC configuration. The default value is `{"ssl": true}`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `jdbc.BlockingRowsMode`                       | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.DisableIsValidQuery`                    | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.DSILogLevel`                            | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.FilterLevel`                            | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.loginTimeout`                           | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.loglevel`                               | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.socketTimeout`                          | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.ssl`                                    | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.sslMode`                                | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.sslRootCert`                            | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.tcpKeepAlive`                           | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.TCPKeepAliveMinutes`                    | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

## Snowflake Loader `storage` section

| Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`                                        | Optional. The only valid value is the default: `snowflake`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `snowflakeRegion`                             | Required. AWS Region used by Snowflake to access its endpoint.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `username`                                    | Required. Snowflake user with necessary role granted to load data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `role`                                        | Optional. Snowflake role with permission to load data. If it is not provided, the default role in Snowflake will be used.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `jdbcAuth.*` (since 6.6.0)                    | Required. JDBC authentication configuration. Supports two modes: `password` and `privateKey`. The legacy root-level `password` field is also accepted for backward compatibility.                                                                                                                                                                                                                                                                                                                                                                                                                |
| `jdbcAuth.type` (since 6.6.0)                 | Required. The authentication type. Possible values: `password` and `privateKey`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `jdbcAuth.password` (since 6.6.0)             | Required if `jdbcAuth.type` is `password`. Password of the Snowflake user. Can be plain text, or read from the EC2 Parameter Store, GCP Secret Manager, or Azure Key Vault.                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `jdbcAuth.privateKey` (since 6.6.0)           | Required if `jdbcAuth.type` is `privateKey`. PEM-formatted unencrypted PKCS8 private key (output of `openssl pkcs8 -topk8 -nocrypt`). Include the full PEM string with BEGIN/END headers. Can be a plain string or fetched from a secrets store.                                                                                                                                                                                                                                                                                                                                                 |
| `account`                                     | Required. Target Snowflake account.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `warehouse`                                   | Required. Snowflake warehouse which the SQL statements submitted by Snowflake Loader will run on.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `database`                                    | Required. Snowflake database which the data will be loaded to.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `schema`                                      | Required. Target schema                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `transformedStage.*`                          | Required if `NoCreds` is chosen as load auth method. Snowflake stage for transformed events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `transformedStage.name`                       | Required if `transformedStage` is included. The name of the stage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `transformedStage.location`                   | Required if `transformedStage` is included. The S3 path used as stage location. (Not needed since 5.2.0 because it is auto-configured)                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `folderMonitoringStage.*`                     | Required if `monitoring.folders` section is configured and `NoCreds` is chosen as load auth method. Snowflake stage to load folder monitoring entries into temporary Snowflake table.                                                                                                                                                                                                                                                                                                                                                                                                            |
| `folderMonitoringStage.name`                  | Required if `folderMonitoringStage` is included. The name of the stage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `folderMonitoringStage.location`              | Required if `folderMonitoringStage` is included. The S3 path used as stage location. (Not needed since 5.2.0 because it is auto-configured)                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `appName`                                     | Optional. Name passed as 'application' property while creating Snowflake connection. The default is `Snowplow_OSS`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `maxError`                                    | Optional. A table copy statement will skip an input file when the number of errors in it exceeds the specified number. This setting is used during initial loading and thus can filter out only invalid JSONs (which is impossible situation if used with Transformer).                                                                                                                                                                                                                                                                                                                          |
| `jdbcHost`                                    | Optional. Host for the JDBC driver that has priority over automatically derived hosts. If it is not given, host will be created automatically according to given `snowflakeRegion`.                                                                                                                                                                                                                                                                                                                                                                                                              |
| `loadAuthMethod.*`                            | Optional, default method is `NoCreds`. Specifies the auth method to use with `COPY INTO` statement. Note that `TempCreds` auth method doesn't work when data is loaded from GCS.                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `loadAuthMethod.type`                         | Required if `loadAuthMethod` section is included. Specifies the type of the auth method. The possible values are `NoCreds` and `TempCreds`. With `NoCreds`, no credentials will be passed to `COPY INTO` statement. Instead, `transformedStage` and `folderMonitoringStage` specified above will be used. More information can be found [here](https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html). With `TempCreds`, temporary credentials will be created for every load operation and these temporary credentials will be passed to `COPY INTO` statement. |
| `loadAuthMethod.roleArn`                      | Required if `loadAuthMethod.type` is `TempCreds`. IAM role that is used while creating temporary credentials. This role should allow access to the S3 bucket the transformer will write data to. You can find the list of necessary permissions needs to be given to role in [here](https://docs.snowflake.com/en/user-guide/data-load-s3-config-aws-iam-user.html).                                                                                                                                                                                                                             |
| `loadAuthMethod.credentialsTtl` since (5.4.0) | Optional, default value `1 hour`. If `TempCreds` load auth method is used, this value will be used as a session duration of temporary credentials used for loading data and folder monitoring. In that case, it can't be greater than 1 hour and can't be less than 15 minutes.                                                                                                                                                                                                                                                                                                                  |
| `readyCheck` since (5.4.0)                    | Optional. Either `ResumeWarehouse` (the default) or `Select1`. The command the loader runs to prepare the JDBC connection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

## Databricks Loader `storage` section

| Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                                        | Optional. The only valid value is the default: `databricks`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `host`                                        | Required. Hostname of Databricks cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `password`                                    | Required if `oauth` section is not defined. [Databricks access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html). Can be plain text, read from the EC2 parameter store or GCP secret manager (see below).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `oauth.clientId`                              | Required if `password` is not defined. Client id used in [Databricks OAuth machine-to-machine](https://docs.databricks.com/en/integrations/jdbc/authentication.html#oauth-machine-to-machine-m2m-authentication) authentication flow.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `oauth.clientSecret`                          | Required if `password` is not defined. Client secret used in [Databricks OAuth machine-to-machine](https://docs.databricks.com/en/integrations/jdbc/authentication.html#oauth-machine-to-machine-m2m-authentication) authentication flow.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `eventsOptimizePeriod`                        | Optional. The default value is `2 days`. Optimize period per table, that will be used as predicate for the `OPTIMIZE` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `password.secretManager.parameterName`        | Alternative way for passing in the access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `schema`                                      | Required. Target schema.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `port`                                        | Required. Port of Databricks cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `httpPath`                                    | Required. Http Path of Databricks cluster. Get it from the JDBC connection details after the cluster has been created.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `catalog`                                     | Optional. The default value is `hive_metastore`. [Databricks unity catalog name](https://docs.databricks.com/data-governance/unity-catalog/index.html).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `userAgent`                                   | Optional. The default value is `snowplow-rdbloader-oss`. User agent name for Databricks connection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `loadAuthMethod.*`                            | Optional, default method is `NoCreds`. Specifies the auth method to use with `COPY INTO` statement                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `loadAuthMethod.type`                         | Required if `loadAuthMethod` section is included. Specifies the type of the auth method. The possible values are `NoCreds` and `TempCreds`. With `NoCreds`, no credentials will be passed to `COPY INTO` statement. Databricks cluster needs to have permission to access transformer output S3 bucket. More information can be found [here](https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html). With `TempCreds`, temporary credentials will be created for every load operation and these temporary credentials will be passed to `COPY INTO` statement. With this way, Databricks cluster doesn't need permission to access to transformer output S3 bucket. This access will be provided by temporary credentials. |
| `loadAuthMethod.roleArn`                      | Required if `loadAuthMethod.type` is `TempCreds`. IAM role that is used while creating temporary credentials. This role should allow access to the S3 bucket the transformer will write data to, with the following permissions: `s3:GetObject*`, `s3:ListBucket`, and `s3:GetBucketLocation`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `loadAuthMethod.credentialsTtl` since (5.4.0) | Optional, default value `1 hour`. If `TempCreds` load auth method is used, this value will be used as a session duration of temporary credentials used for loading data and folder monitoring. In that case, it can't be greater than 1 hour and can't be less than 15 minutes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `logLevel` since (5.3.2)                      | Optional. The default value is 3. Specifies JDBC driver log level. 0 - Disable all logging. 1 - Log severe error events that lead the driver to abort. 2 - Log error events that might allow the driver to continue running. 3 - Log events that might result in an error if action is not taken. (default) 4 - Log general information that describes the progress of the driver. 5 - Log detailed information that is useful for debugging the driver. 6 - Log all driver activity.                                                                                                                                                                                                                                                                                 |

## AWS specific settings

| Parameter                | Description                                                                                                                                                                                                          |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `region`                 | Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html). AWS region of the S3 bucket. |
| `messageQueue.type`      | Required. Type of the message queue. It should be `sqs` when application is run on AWS.                                                                                                                              |
| `messageQueue.queueName` | Required. The name of the SQS queue used by the transformer and loader to communicate.                                                                                                                               |
| `jsonpaths`              | Optional. An S3 URI that holds JSONPath files.                                                                                                                                                                       |

## GCP specific settings

Only Snowflake Loader can be run on GCP at the moment.

| Parameter                   | Description                                                                                      |
| --------------------------- | ------------------------------------------------------------------------------------------------ |
| `messageQueue.type`         | Type of the message queue. It should be `pubsub` when application is run on GCP.                 |
| `messageQueue.subscription` | Required. The name of the Pubsub subscription used by the transformer and loader to communicate. |

## Azure specific settings

Only Snowflake Loader can be run on Azure at the moment.

| Parameter                       | Description                                                                                                                                                                |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `blobStorageEndpoint`           | Endpoint of Azure Blob Storage container that contains transformer's output.                                                                                               |
| `azureVaultName`                | Name of the Azure Key Vault where application secrets are stored. Required if secret store is used in `storage.jdbcAuth.password` or `storage.jdbcAuth.privateKey` fields. |
| `messageQueue.type`             | Type of the message queue. It should be `kafka` when application is run on Azure.                                                                                          |
| `messageQueue.topicName`        | Name of the Kafka topic used to communicate with Transformer.                                                                                                              |
| `messageQueue.bootstrapServers` | A list of host:port pairs to use for establishing the initial connection to the Kafka cluster.                                                                             |
| `messageQueue.consumerConf`     | Optional. Kafka consumer configuration. See <https://kafka.apache.org/documentation/#consumerconfigs> for all properties.                                                  |

## Common loader settings

| Parameter                            | Description                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schedules.*`                        | Optional. Tasks scheduled to run periodically.                                                                                                                                                                                                                                                                                                                                                            |
| `schedules.noOperation.[*]`          | Optional. Array of objects which specifies no-operation windows during which periodically scheduled tasks (configured in this section) will not run.                                                                                                                                                                                                                                                      |
| `schedules.noOperation.[*].name`     | Human-readable name of the no-op window.                                                                                                                                                                                                                                                                                                                                                                  |
| `schedules.noOperation.[*].when`     | Cron expression with second granularity.                                                                                                                                                                                                                                                                                                                                                                  |
| `schedules.noOperation.[*].duration` | For how long the loader should be paused.                                                                                                                                                                                                                                                                                                                                                                 |
| `schedules.optimizeEvents`           | Optional. The default value is `"0 0 0 ? * *"` (i.e. every day at 00:00, JVM timezone). Cron expression with second granularity that specifies the schedule to run periodically an `OPTIMIZE` statement on event table. (Only for Databricks Loader)                                                                                                                                                      |
| `schedules.optimizeManifest`         | Optional. The default value is `"0 0 5 ? * *"` (i.e. every day at 05:00 AM, JVM timezone). Cron expression with second granularity that specifies the schedule to run periodically an `OPTIMIZE` statement on manifest table. (Only for Databricks Loader)                                                                                                                                                |
| `retryQueue.*`                       | Optional. Additional backlog of recently failed folders that could be automatically retried. Retry queue saves a failed folder and then re-reads the info from `shredding_complete` S3 file. (Despite the legacy name of the message, which is required for backward compatibility, this also works with wide row format data.)                                                                           |
| `retryQueue.period`                  | Required if `retryQueue` section is configured. How often batch of failed folders should be pulled into a discovery queue.                                                                                                                                                                                                                                                                                |
| `retryQueue.size`                    | Required if `retryQueue` section is configured. How many failures should be kept in memory. After the limit is reached new failures are dropped.                                                                                                                                                                                                                                                          |
| `retryQueue.maxAttempts`             | Required if `retryQueue` section is configured. How many attempts to make for each folder. After the limit is reached new failures are dropped.                                                                                                                                                                                                                                                           |
| `retryQueue.interval`                | Required if `retryQueue` section is configured. Artificial pause after each failed folder being added to the queue.                                                                                                                                                                                                                                                                                       |
| `retries.*`                          | Optional. Unlike `retryQueue` these retries happen immediately, without proceeding to another message.                                                                                                                                                                                                                                                                                                    |
| `retries.backoff`                    | Required if `retries` section is configured. Starting backoff period, eg '30 seconds'.                                                                                                                                                                                                                                                                                                                    |
| `retries.strategy`                   | Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`.                                                                                                                                                                                                                                                                                             |
| `retries.attempts`                   | Optional. How many attempts to make before sending the message into retry queue. If missing, `cumulativeBound` will be used.                                                                                                                                                                                                                                                                              |
| `retries.cumulativeBound`            | Optional. When backoff reaches this delay, eg '1 hour', the loader will stop retrying. If both this and `attempts` are not set, the loader will retry indefinitely.                                                                                                                                                                                                                                       |
| `timeouts.loading`                   | Optional. How long, eg '1 hour', `COPY` statement execution can take before considering Redshift unhealthy. If no progress (ie, moving to a different subfolder) within this period, the loader will abort the transaction.                                                                                                                                                                               |
| `timeouts.nonLoading`                | Optional. How long, eg '10 mins', non-loading steps such as `ALTER TABLE` can take before considering Redshift unhealthy.                                                                                                                                                                                                                                                                                 |
| `timeouts.sqsVisibility`             | Optional. The time window in which a message must be acknowledged. Otherwise it is considered abandoned. If a message has been pulled, but hasn't been acked, the time before it is again available to consumers is equal to this, eg '5 mins'. Another consequence is that if the loader has failed on processing a message, the next time it will get this (or anything) from the queue has this delay. |
| `readyCheck.*`                       | Optional. Check the target destination to make sure it is ready.                                                                                                                                                                                                                                                                                                                                          |
| `readyCheck.backoff`                 | Optional. The default value is `15 seconds`. Starting backoff period.                                                                                                                                                                                                                                                                                                                                     |
| `readyCheck.strategy`                | Optional. The default value is `CONSTANT`. Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`.                                                                                                                                                                                                                                                  |
| `initRetries.*`                      | Optional. Retries configuration for initialization block. It will retry on all exceptions from there.                                                                                                                                                                                                                                                                                                     |
| `initRetries.backoff`                | Required if `initRetries` section is configured. Starting backoff period, eg '30 seconds'.                                                                                                                                                                                                                                                                                                                |
| `initRetries.strategy`               | Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`.                                                                                                                                                                                                                                                                                             |
| `initRetries.attempts`               | Optional. How many attempts to make before sending the message into retry queue. If missing, `cumulativeBound` will be used.                                                                                                                                                                                                                                                                              |
| `initRetries.cumulativeBound`        | Optional. When backoff reaches this delay, eg '1 hour', the loader will stop retrying. If both this and `attempts` are not set, the loader will retry indefinitely.                                                                                                                                                                                                                                       |
| `telemetry.disable`                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                 |
| `telemetry.userProvidedId`           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                       |

## Common monitoring settings

| Parameter                              | Description                                                                                                                                                                                                       |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `monitoring.webhook.endpoint`          | Optional. An HTTP endpoint where monitoring alerts should be sent.                                                                                                                                                |
| `monitoring.webhook.tags`              | Optional. Custom key-value pairs which can be added to the monitoring webhooks. Eg, `{"tag1": "label1"}`.                                                                                                         |
| `monitoring.snowplow.appId`            | Optional. When using Snowplow tracking, set this `appId` in the event.                                                                                                                                            |
| `monitoring.snowplow.collector`        | Optional. Set to a collector URL to turn on Snowplow tracking.                                                                                                                                                    |
| `monitoring.sentry.dsn`                | Optional. For tracking runtime exceptions.                                                                                                                                                                        |
| `monitoring.metrics.*`                 | Send metrics to a StatsD server or stdout.                                                                                                                                                                        |
| `monitoring.metrics.period`            | Optional. The default is 5 minutes. Period for metrics emitted periodically.                                                                                                                                      |
| `monitoring.metrics.statsd.*`          | Optional. For sending loading metrics (latency and event counts) to a StatsD server.                                                                                                                              |
| `monitoring.metrics.statsd.hostname`   | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                |
| `monitoring.metrics.statsd.port`       | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                         |
| `monitoring.metrics.statsd.tags`       | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                      |
| `monitoring.metrics.statsd.prefix`     | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.rdbloader`.                                                                                                                       |
| `monitoring.metrics.stdout.*`          | Optional. For sending metrics to stdout.                                                                                                                                                                          |
| `monitoring.metrics.stdout.prefix`     | Optional. Overrides the default metric prefix.                                                                                                                                                                    |
| `monitoring.folders.*`                 | Optional. Configuration for periodic unloaded / corrupted folders checks.                                                                                                                                         |
| `monitoring.folders.staging`           | Required if `monitoring.folders` section is configured. Path where loader could store auxiliary logs for folder monitoring. Loader should be able to write here, storage target should be able to load from here. |
| `monitoring.folders.period`            | Required if `monitoring.folders` section is configured. How often to check for unloaded / corrupted folders.                                                                                                      |
| `monitoring.folders.since`             | Optional. Specifies from when folder monitoring will start to monitor. Note that this is a duration, eg `7 days`, relative to when the loader is launched.                                                        |
| `monitoring.folders.until`             | Optional. Specifies until when folder monitoring will monitor. Note that this is a duration, eg `7 days`, relative to when the loader is launched.                                                                |
| `monitoring.folders.transformerOutput` | Required if `monitoring.folders` section is configured. Path to transformed archive.                                                                                                                              |
| `monitoring.folders.failBeforeAlarm`   | Required if `monitoring.folders` section is configured. How many times the check can fail before generating an alarm. Within the specified tolerance, failures will log a `WARNING` instead.                      |
| `monitoring.healthCheck.*`             | Optional. Periodic DB health check, raising a warning if DB hasn't responded to `SELECT 1`.                                                                                                                       |
| `monitoring.healthCheck.frequency`     | Required if `monitoring.healthCheck` section is configured. How often to run a periodic DB health check.                                                                                                          |
| `monitoring.healthCheck.timeout`       | Required if `monitoring.healthCheck` section is configured. How long to wait for a health check response.                                                                                                         |

---

# Event deduplication with Spark transformer
> Deduplicate Snowplow events in-batch and cross-batch using natural and synthetic deduplication strategies with DynamoDB storage.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/

**NOTE:** Deduplication is currently only available in the [Spark transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/).

Duplicates are a common problem in event pipelines. At the root of it is the fact that we can't guarantee every event has a unique UUID because:

- We have no exactly-once delivery guarantees
- User-side software can send events more than once
- Robots can send events reusing the same event ID

Depending on your use case, you may choose to ignore duplicates, or deal with them once the events are in the data warehouse.

If you are loading into **Redshift** (using [shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data)), we strongly recommend to deduplicate the data upstream of loading. Once duplicates are loaded into separate tables, table joins would create a Cartesian product.

This is less of a concern with [wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format) loading into **Snowflake**, where it's easier to deduplicate during the data modeling step in the warehouse.

This table shows the available deduplication mechanisms:

| Strategy                            | Batch?      | Same event ID? | Same event fingerprint? | Availability      |
| ----------------------------------- | ----------- | -------------- | ----------------------- | ----------------- |
| In-batch natural deduplication      | In-batch    | Yes            | Yes                     | Spark transformer |
| In-batch synthetic deduplication    | In-batch    | Yes            | No                      | Spark transformer |
| Cross-batch natural deduplication   | Cross-batch | Yes            | Yes                     | Spark transformer |
| Cross-batch synthetic deduplication | Cross-batch | Yes            | No                      | Not supported     |

## In-batch natural deduplication

"Natural duplicates" are events which share the same event ID (`event_id`) and the same event payload (`event_fingerprint`), meaning that they are semantically identical to each other. For a given batch of events being processed, RDB Transformer keeps only the first out of each group of natural duplicates and discards all others.

To enable this functionality, you need to have the [Event Fingerprint Enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/) enabled in Enrich. This will correctly populate the `event_fingerprint` property. No changes are required in the transformer's own `config.hocon` file.

If the fingerprint enrichment is not enabled, the transformer will assign a random UUID to each event, effectively marking all events as non-duplicates (in the 'natural' sense).

## In-batch synthetic deduplication

"Synthetic duplicates" are events which share the same event ID (`event_id`), but have different event payload (`event_fingerprint`), meaning that they can be either semantically independent events or the same events with slightly different payloads (caused by third-party software). For a given batch of events being processed, RDB Transformer uses the following strategy:

- Collect all the events with identical `event_id` which are left after natural deduplication
- Generate new random `event_id` for each of them
- Create a [`duplicate`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/duplicate/jsonschema/1-0-0) context with the original `event_id` for each event where the duplicated `event_id` was found.

There is no transformer configuration required for this functionality: deduplication is performed automatically. It is optional but highly recommended to use the [Event Fingerprint Enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/) in Enrich in order to correctly populate the `event_fingerprint` property.

## Cross-batch natural deduplication

The strategies described above deal with duplicates within the same batch of data being processed. But what if events are duplicated across batches?

To apply any of these strategies, we need to store information about previously seen duplicates, so that we can compare events in the current batch against them. We don't need to store the whole event: just the `event_id` and the `event_fingerprint` fields.

We need to store these in a database that allows fast random access, so we chose DynamoDB, a fully managed NoSQL database service.

### How to enable cross-batch natural deduplication

To enable cross-batch natural deduplication, you must provide a third configuration option in the `RDB Transformer` step of the Dataflow Runner playbook, using the `--duplicate-storage-config` flag. Like the other options, this needs to be provided as a base64-encoded string. This config file contains information about the DynamoDB table to be used, as well as credentials for accessing it. For more details on the config file structure, refer to the [Snowplow Events Manifest](https://github.com/snowplow-incubator/snowplow-events-manifest) library and its documentation.

An example step definition can look like this:

```json
{

  "type": "CUSTOM_JAR",

  "name": "RDB Transformer",

  "actionOnFailure": "CANCEL_AND_WAIT",

  "jar": "command-runner.jar",

  "arguments": [

    "spark-submit",

    "--class", "com.snowplowanalytics.snowplow.rdbloader.shredder.batch.Main",

    "--master", "yarn",

    "--deploy-mode", "cluster",

    "s3://snowplow-hosted-assets-eu-central-1/4-storage/transformer-batch/snowplow-transformer-batch-4.1.0.jar",

    "--iglu-config", "{{base64File "/home/snowplow/configs/snowplow/iglu_resolver.json"}}",

    "--config", "{{base64File "/home/snowplow/configs/snowplow/config.hocon"}}",

    "--duplicate-storage-config", "{{base64File "/home/snowplow/configs/snowplow/duplicate-storage-config.json"}}"

  ]

}
```

If this configuration option is not provided, cross-batch natural deduplication will be disabled. In-batch deduplication will still work however.

### Costs and performance implications

Cross-batch deduplication uses DynamoDB as transient storage and therefore has associated AWS costs. The default write capacity is 100 units, which should roughly cost USD50 per month. Note that at this rate your shred job can get throttled by insufficient capacity, even with a very powerful EMR cluster. You can tweak throughput to match your needs but that will inflate the bill.

### How RDB Transformer uses DynamoDB for deduplication

We store duplicate data in a DynamoDB table with the following attributes:

- `eventId`, a String
- `fingerprint`, a String
- `etlTime`, a Date
- `ttl`, a Date.

We can query this table to see if the event that is currently being processed has been seen before based on `event_id` and `event_fingerprint`.

We store the `etl_timestamp` to prevent issues in case of a failed transformer run. If a run fails and is then rerun, we don't want the rerun to consider rows in the DynamoDB table which were written as part of the failed run. Otherwise all events that were processed by the failed run will be rejected as duplicates.

To update the DynamoDB table, RDB Transformer uses so-called 'conditional updates' to perform a check-and-set operation on a per-event basis. The algorithm is as follows:

- Attempt to write the `(event_id, event_fingerprint, etl_timestamp)` triple to DynamoDB but succeed only if the `(event_id, event_fingerprint)` pair cannot be found in the table with an earlier `etl_timestamp` than the current one.
- If the write fails, we have a natural duplicate. We can safely drop it because we know that we have the 'original' of this event already safely in the data warehouse.
- If the write succeeds, we know we have an event which is not a natural duplicate. (It could still be a synthetic duplicate however.)

The transformer performs this check after grouping the batch by `event_id` and `event_fingerprint`. This ensures that all check-and-set requests for a specific `(event_id, event_fingerprint)` pair will come from a single mapper, avoiding race conditions.

To keep the DynamoDB table size in check, we're using the time-to-live feature which provides automatic cleanup after the specified time. For event manifests this time is the ETL timestamp plus 180 days. This is stored in the table's `ttl` attribute.

### Creating the DynamoDB table and IAM policy

If you provide a `duplicate-storage-config` that specifies a DynamoDB table but RDB Transformer can't find it upon launch, it will create it with the default provisioned throughput. That might not be enough for the amount of data you want to process. Creating the table upfront gives you the opportunity to spec it out according to your needs. This step is optional but recommended.

1. The table name can be anything, but it must be unique.
2. The partition key must be called `eventId` and have type String. The sort key must be called `fingerprint` and have type String. You can refer to the the [DynamoDB table definition](#how-rdb-transformer-uses-dynamodb-for-deduplication) above for the full table schema.
3. Uncheck the "Use default settings" checkbox and set "Write capacity units" to 100 (or your desired value).
4. After the table is created, note down its ARN in the "Overview" tab.
5. Create the IAM policy In the AWS console, navigate to IAM and go to "Policies". Select "Create Your Own Policy" and choose a descriptive name. Here's an example Policy Document that you can paste:

```json
{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Sid": "Stmt1486765706000",

      "Effect": "Allow",

      "Action": [

        "dynamodb:CreateTable",

        "dynamodb:DeleteTable",

        "dynamodb:DescribeTable",

        "dynamodb:PutItem"

      ],

      "Resource": [

        "arn:aws:dynamodb:us-east-1:{{AWS_ACCOUNT_ID}}:table/snowplow-deduplication"

      ]

    }

  ]

}
```

Replace the element in the `Resources` array with the ARN that you noted down in step 4. If you've already created the table, the policy does not require the `dynamodb:CreateTable` and `dynamodb:DeleteTable` permissions.

---

# How the RDB Loader transforms enriched data for warehouses
> Transform Snowplow enriched events into shredded data for Redshift or wide row format for Snowflake and Databricks using Spark or stream transformers.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/

_For a high-level overview of the RDB Loader architecture, of which the transformer is a part, see [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)._

The `transformer` application can have two types of output:

- 'shredded' data
- wide row format.

Both the Spark transformer and the stream transformer can output both types. Which type to pick, depends on the intended storage target.

For loading into **Redshift**, use [shredded data](#shredded-data).

For loading into **Snowflake**, use [wide row format](#wide-row-format).

For loading into **Databricks**, use [wide row format](#wide-row-format).

## Shredded data

Shredding is the process of splitting a Snowplow enriched event into several smaller chunks, which can be inserted directly into **Redshift** tables.

A Snowplow enriched event is a 131-column tsv line. Each line contains all fields that constitute a specific event, including its id, timestamps, custom and derived contexts, etc.

After shredding, the following entities are split out from the original event:

1. **Atomic event.** A tsv line very similar to the enriched event but not containing JSON fields (`contexts`, `derived_contexts` and `unstruct_event`). The results are stored under a path similar to `shredded/good/run=2016-11-26-21-48-42/atomic-events/part-00000` and are available to load with RDB Loader or directly with Redshift `COPY`.
2. **Contexts.** Two JSON fields -- `contexts` and `derived_contexts` -- are extracted from the enriched event. Their original values are validated self-describing JSONs, consisting of a `schema` and a `data` property. After shredding, a third property is added, called `hierarchy`. This `hierarchy` contains fields you can use to later join your context SQL tables with the `atomic.events` table. One atomic event can be associated with multiple context entities. The results are stored under a path like `shredded/good/run=2016-11-26-21-48-42/shredded-types/vendor=com.acme/name=my_context/format=jsonschema/version=1-0-1/part-00000`, where the `part-*` files are valid ndJSON files which can be loaded with RDB Loader or directly with Redshift `COPY`.
3. **Self-describing (unstructured) events.** Same as the contexts described above but there is a strict one-to-one relation with atomic events. The results are stored under a path with the same structure as for contexts and are ready to be loaded with RDB Loader or directly with Redshift `COPY`.

These files are stored on S3 partitioned by type. When the data is loaded into Redshift, each type goes to a dedicated table.

The following diagram illustrates the process:

![Diagram of the shredding process: an enriched good event TSV file with contexts, unstruct\_event, and derived\_contexts fields is passed through a Shredding step (using Iglu for schema lookup), producing shredded good output split into atomic.events, user-specific self-describing events (JSON), and predefined contexts (JSON), which are then loaded by StorageLoader into Redshift tables](/assets/images/storage-loader-dataflow-96341b5e426da988ea3bc5c07a4949d7.png)

**NOTE:** Shredded data can currently only be loaded into **Redshift**.

## Wide row format

Unlike shredding, wide row format preserves data as a single line per event, with one column for each different type of contexts and self-describing events.

For contexts (aka entities), the type of the column is `ARRAY` and the name looks like `contexts_com_acme_my_custom_entity_schema_1`. Note the plural that matches the `ARRAY` type: in theory each `contexts_*` column may contain multiple entities.

For self-describing events, the type of the column is `OBJECT` and the name looks like `unstruct_event_com_acme_my_custom_event_schema_1`. Each line in the table contains only 1 event.

The values in these columns have recursive structure with arbitrary depth, which depends on the schema that describes them.

The results are stored under a path like `output=good/part-00000` and can be loaded with RDB Loader.

There are two options as output file format with wide row transformation: JSON and Parquet

JSON file format can be used for loading into **Snowflake** and Parquet file format can be used for loading into **Databricks**

---

# Spark transformer configuration reference
> Configure Spark batch transformer with EMR settings, S3 paths, output formats, and deduplication options for warehouse loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/configuration-reference/

The configuration reference in this page is written for Spark Transformer `6.6.0`

An example of the minimal required config for the Spark transformer can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

| Parameter                                                       | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input`                                                         | Required. S3 URI of the enriched archive. It must be populated separately with `run=YYYY-MM-DD-hh-mm-ss` directories.                                                                                                                                                                                                                                                                                         |
| `runInterval.*`                                                 | Specifies interval to process.                                                                                                                                                                                                                                                                                                                                                                                |
| `runInterval.sinceAge`                                          | Optional. A duration that specifies the maximum age of folders that should get processed. If `sinceAge` and `sinceTimestamp` are both specified, then the latest value of the two determines the earliest folder that will be processed.                                                                                                                                                                      |
| `runInterval.until`                                             | Optional. Process until this timestamp.                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.sentry.dsn`                                         | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                    |
| `monitoring.metrics.cloudwatch` (since 5.5.0)                   | Optional. For sending metrics to Cloudwatch. If not set, metrics are not sent.                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.cloudwatch.namespace` (since 5.5.0)         | Namespace that will contain the metrics in Cloudwatch. Example: `snowplow/transformer_batch`                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.cloudwatch.transformDuration` (since 5.5.0) | Name of the metric that contains the number of milliseconds needed to transform a folder. Example: `transform_duration`                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.cloudwatch.dimensions` (since 5.5.0)        | Any key-value pairs to be added as dimensions in Cloudwatch metrics. Example:```json
{"app_version": "x.y.z", "env": "prod"}
```                                                                                                                                                                                                                                                                              |
| `deduplication.*`                                               | Configure the way in-batch deduplication is performed                                                                                                                                                                                                                                                                                                                                                         |
| `deduplication.synthetic.type`                                  | Optional. The default is `BROADCAST`. Can be `NONE` (disable), `BROADCAST` (default) and `JOIN` (different low-level implementations).                                                                                                                                                                                                                                                                        |
| `deduplication.synthetic.cardinality`                           | Optional. The default is 1. Do not deduplicate pairs with less-or-equal cardinality.                                                                                                                                                                                                                                                                                                                          |
| `deduplication.natural`                                         | Optional. The default is 'true'. Enable or disable natural deduplication. Available since `5.1.0`                                                                                                                                                                                                                                                                                                             |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0)            | Optional, default = false. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                                                                                                                                              |
| `skipSchemas` (since 5.7.1)                                     | Optional, default = none. Supply a list of Iglu URIs and the transformer's output files will omit any columns using that schema. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                                                                                                                    |
| `archival.source` (since 6.5.0)                                 | Optional. An S3 URI holding enriched data to archive. When set, the transformer copies all files from this path into a new `run=YYYY-MM-DD-hh-mm-ss` directory under `input` before processing begins, then deletes each source file. Omit it (or leave the `archival` block empty) to skip the archival step. This can replace the standalone `s3-dist-cp` EMR step.                                         |
| `output.path`                                                   | Required. S3 URI of the transformed output.                                                                                                                                                                                                                                                                                                                                                                   |
| `output.compression`                                            | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                                                                                                                                     |
| `output.region`                                                 | AWS region of the S3 bucket. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                          |
| `output.maxRecordsPerFile` (since 5.4.0)                        | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.type` (since 5.4.0)                                 | Optional. Either `kinesis` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                                                                                                                                   |
| `output.bad.streamName` (since 5.4.0)                           | Required if output type is `kinesis`. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                                                                                                                 |
| `output.bad.region` (since 5.4.0)                               | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `output.bad.recordLimit` (since 5.4.0)                          | Optional, default = 500. Limits the number of events in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                                  |
| `output.bad.byteLimit` (since 5.4.0)                            | Optional, default = 5242880. Limits the number of bytes in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                               |
| `output.bad.backoffPolicy.minBackoff` (since 5.4.0)             | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                     |
| `output.bad.backoffPolicy.maxBackoff` (since 5.4.0)             | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                           |
| `output.bad.backoffPolicy.maxRetries` (since 5.4.0)             | Optional, default = 10. Maximum number of retries for internal Kinesis errors.                                                                                                                                                                                                                                                                                                                                |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 5.4.0)    | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails in case of throughput exceeded.                                                                                                                                                                                                                                                                           |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 5.4.0)    | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails in case of throughput exceeded. Writing is retried forever.                                                                                                                                                                                                                                                     |
| `queue.type`                                                    | Required. Type of the message queue. Can be either `sqs` or `sns`.                                                                                                                                                                                                                                                                                                                                            |
| `queue.queueName`                                               | Required if queue type is `sqs`. Name of the SQS queue. SQS queue needs to be FIFO.                                                                                                                                                                                                                                                                                                                           |
| `queue.topicArn`                                                | Required if queue type is `sns`. ARN of the SNS topic.                                                                                                                                                                                                                                                                                                                                                        |
| `queue.region`                                                  | AWS region of the SQS queue or SNS topic. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                             |
| `formats.*`                                                     | Schema-specific format settings.                                                                                                                                                                                                                                                                                                                                                                              |
| `formats.transformationType`                                    | Required. Type of transformation, either `shred` or `widerow`. See [Shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) and [Wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format).                                                                             |
| `formats.fileFormat`                                            | Optional. The default is `JSON`. Output file format produced when transformation is `widerow`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                                                                                                    |
| `formats.default`                                               | Optional. The default is `TSV`. Data format produced by default when transformation is `shred`. Either `TSV` or `JSON`. `TSV` is recommended as it enables table autocreation, but requires an Iglu Server to be available with known schemas (including Snowplow schemas). `JSON` does not require an Iglu Server, but requires Redshift JSONPaths to be configured and does not support table autocreation. |
| `formats.tsv`                                                   | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `JSON` this list of schemas will still be shredded into `TSV`.                                                                                                                                                                                                                                    |
| `formats.json`                                                  | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `TSV` this list of schemas will still be shredded into `JSON`.                                                                                                                                                                                                                                    |
| `formats.skip`                                                  | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. Schemas for which loading can be skipped.                                                                                                                                                                                                                                                                                |
| `validations.*`                                                 | Optional. Criteria to validate events against                                                                                                                                                                                                                                                                                                                                                                 |
| `validations.minimumTimestamp`                                  | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                                                                                                                                            |
| `featureFlags.*`                                                | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                                                                                                                                                   |
| `featureFlags.legacyMessageFormat`                              | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                                                                                                                                          |
| `featureFlags.truncateAtomicFields` (since 5.4.0)               | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                                                                                                                                                  |

---

# Spark transformer for batch processing
> Transform enriched Snowplow data in batches using Spark on EMR with support for deduplication and wide row or shredded output.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/

> **Info:** For a high-level overview of the Transform process, see [Transforming enriched data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/). For guidance on picking the right `transformer` app, see [How to pick a transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/#how-to-pick-a-transformer).

The Spark-based transformer is a batch job designed to be deployed in an EMR cluster and process a bounded data set stored on S3.

In order to run it, you will need:

- the `snowplow-transformer-batch` jar file (from version 3.0.0 this replaces the `snowplow-rdb-shredder` asset)
- configuration files for the jar file
- an EMR cluster specification
- a way to spin up an EMR cluster and submit a job to it.

You can use any suitable tool to periodically submit the transformer job to an EMR cluster. We recommend you use our purpose-built [Dataflow Runner](https://github.com/snowplow/dataflow-runner) tool. All the examples below assume that Dataflow Runner is being used. Refer to the app's [documentation](/docs/api-reference/dataflow-runner/) for more details.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available in several S3 buckets that are accessible to an EMR cluster:

```text
s3://snowplow-hosted-assets/4-storage/transformer-batch/snowplow-transformer-batch-6.6.0.jar





-- or --





s3://snowplow-hosted-{{ region }}/4-storage/transformer-batch/snowplow-transformer-batch-6.6.0.jar
```

where `region` is one of `us-east-1`, `us-east-2`, `us-west-1`, `us-west-2`, `eu-central-1`, `eu-west-2`, `ca-central-1`, `sa-east-1`, `ap-southeast-1`, `ap-southeast-2`, `ap-northeast-1`, `ap-northeast-2`, or `ap-south-1`. Pick the region of your EMR cluster.

## Configuring the EMR cluster

> **Note:** Starting from version `5.5.0`, batch transformer requires to use Java 11 on EMR ([default is Java 8](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/configuring-java8.html)). See the `bootstrapActionConfigs` section in the configuration below.

Here's an example of an EMR cluster config file that can be used with Dataflow Runner:

```json
{

  "schema": "iglu:com.snowplowanalytics.dataflowrunner/ClusterConfig/avro/1-1-0",

  "data": {

    "name": "RDB Transformer",

    "region": "eu-central-1",

    "logUri": "s3://com-acme/logs/",

    "credentials": {

      "accessKeyId": "env",

      "secretAccessKey": "env"

    },

    "roles": {

      "jobflow": "EMR_EC2_DefaultRole",

      "service": "EMR_DefaultRole"

    },

    "ec2": {

      "amiVersion": "6.2.0",

      "keyName": "ec2-key-name",

      "location": {

        "vpc": {

          "subnetId": "subnet-id"

        }

      },

      "instances": {

        "master": {

          "type": "m4.large",

          "ebsConfiguration": {

            "ebsOptimized": true,

            "ebsBlockDeviceConfigs": []

          }

        },

        "core": {

          "type": "r4.xlarge",

          "count": 1

        },

        "task": {

          "type": "m4.large",

          "count": 0,

          "bid": "0.015"

        }

      }

    },

    "tags": [],

    "bootstrapActionConfigs": [

      {

        "name": "Use Java 11",

        "scriptBootstrapAction": {

          "path": "s3://<bucket>/<path>/emr-bootstrap-java-11.sh",

          "args": []

        }

      }

    ],

    "configurations": [

      {

        "classification": "core-site",

        "properties": {

          "Io.file.buffer.size": "65536"

        },

        "configurations": []

      },

      {

        "classification": "yarn-site",

        "properties": {

          "yarn.nodemanager.resource.memory-mb": "57344",

          "yarn.scheduler.maximum-allocation-mb": "57344",

          "yarn.nodemanager.vmem-check-enabled": "false"

        },

        "configurations": []

      },

      {

        "classification": "spark",

        "properties": {

          "maximizeResourceAllocation": "false"

        },

        "configurations": []

      },

      {

        "classification": "spark-defaults",

        "properties": {

          "spark.executor.memory": "7G",

          "spark.driver.memory": "7G",

          "spark.driver.cores": "3",

          "spark.yarn.driver.memoryOverhead": "1024",

          "spark.default.parallelism": "24",

          "spark.executor.cores": "1",

          "spark.executor.instances": "6",

          "spark.yarn.executor.memoryOverhead": "1024",

          "spark.dynamicAllocation.enabled": "false"

        },

        "configurations": []

      }

    ],

    "applications": [

      "Hadoop",

      "Spark"

    ]

  }

}
```

You will need to replace `<bucket>` and `<path>` (in the `bootstrapActionConfigs` section) according to where you placed `emr-bootstrap-java-11.sh`. The content of this file should be as follows:

```bash
#!/bin/bash



set -e



sudo update-alternatives --set java /usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/java



exit 0
```

This is a typical cluster configuration for processing \~1.5GB of uncompressed enriched data.

You need to change the following settings with your own values:

- `region`: the AWS region of your EMR cluster
- `logUri`: the location of an S3 bucket where EMR logs will be written
- `ec2.keyName` (optional): The name of an EC2 key pair that you’ll use to shh into the EMR cluster
- `ec2.location.vpc.subnetId`: your VPN subnet ID.

## Configuring `snowplow-transformer-batch`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Spark transformer can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Spark transformer

To run the transformer on EMR with Dataflow Runner, you need:

- the EMR cluster config (see [Configuring the EMR cluster](#configuring-the-emr-cluster) above)
- a Dataflow Runner playbook (a DAG with steps to be submitted to the EMR cluster).

### Preparing the Dataflow Runner playbook

A typical playbook can look like:

```json


{

"schema": "iglu:com.snowplowanalytics.dataflowrunner/PlaybookConfig/avro/1-0-1",

"data": {

  "region": "eu-central-1",

  "credentials": {

    "accessKeyId": "env",

    "secretAccessKey": "env"

  },

  "steps": [

    {

      "type": "CUSTOM_JAR",

      "name": "S3DistCp enriched data archiving",

      "actionOnFailure": "CANCEL_AND_WAIT",

      "jar": "/usr/share/aws/emr/s3-dist-cp/lib/s3-dist-cp.jar",

      "arguments": [

        "--src", "s3://com-acme/enriched/sink/",

        "--dest", "s3://com-acme/enriched/archive/run={{nowWithFormat "2006-01-02-15-04-05"}}/",

        "--s3Endpoint", "s3-eu-central-1.amazonaws.com",

        "--srcPattern", ".*",

        "--outputCodec", "gz",

        "--deleteOnSuccess"

      ]

    },

    {

      "type": "CUSTOM_JAR",

      "name": "RDB Transformer",

      "actionOnFailure": "CANCEL_AND_WAIT",

      "jar": "command-runner.jar",

      "arguments": [

        "spark-submit",

        "--class", "com.snowplowanalytics.snowplow.rdbloader.transformer.batch.Main",

        "--master", "yarn",

        "--deploy-mode", "cluster",

        "s3://snowplow-hosted-assets-eu-central-1/4-storage/transformer-batch/snowplow-transformer-batch-6.6.0.jar",

        "--iglu-config", "{{base64File "/home/snowplow/configs/snowplow/iglu_resolver.json"}}",

        "--config", "{{base64File "/home/snowplow/configs/snowplow/config.hocon"}}"

      ]

    }

  ],

  "tags": []

}

}
```

This playbook consists of two steps. The first one copies the enriched data to a dedicated directory, from which the transformer will read it. The second step is the transformer Spark job that transforms the data.

You need to change the following settings with your own values:

- `region`: the AWS region of the EMR cluster
- `"--src"`: the bucket in which your enriched data is sunk by Enrich
- `"--dest"`: the bucket in which the data for your enriched data lake lives.

**NOTE:** The `"--src"` and `"--dest"` settings above apply only to the `s3DistCp` step of the playbook. The source and destination buckets for the transformer step are configured via the `config.hocon` file.

> **Tip:** Since version 6.5.0, the transformer can perform this copy step itself instead of relying on a separate `s3-dist-cp` step. Set [`archival.source`](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/configuration-reference/) in `config.hocon` to the S3 path of your enriched data. Before processing begins, the transformer copies all files from that path into a new `run=YYYY-MM-DD-hh-mm-ss` directory under `input`, then deletes each source file. When you use this option, you can remove the `S3DistCp` step from the playbook.

### Submitting the job to EMR with Dataflow Runner

Here's an example of putting all of the above together on a transient EMR cluster:

```bash
$ ./dataflow-runner run-transient \

  --emr-config path/to/cluster.conig \

  --emr-playbook path/to/playbook
```

This will spin up the cluster with the above configuration, submit the steps from the playbook, and terminate the cluster once all steps are completed.

For more examples on running EMR jobs with Dataflow Runner, as well as details on cluster configurations and playbooks, see the app's [documentation](/docs/api-reference/dataflow-runner/). It also details how you can submit steps to a persistent EMR cluster.

---

# Stream transformer for real-time processing
> Transform enriched Snowplow data in real-time from Kinesis, Pub/Sub, or Kafka streams without Spark or EMR.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/

> **Info:** For a high-level overview of the Transform process, see [Transforming enriched data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/). For guidance on picking the right `transformer` app, see [How to pick a transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/#how-to-pick-a-transformer).

Unlike the [Spark transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/), the stream transformer reads data directly from the enriched stream and does not use Spark or EMR. It's a plain JVM application, like Stream Enrich or S3 Loader.

Reading directly from stream means that the transformer can bypass the `s3DistCp` staging / archiving step.

Another benefit is that it doesn't process a bounded data set and can emit transformed folders based only on its configured frequency. This means the pipeline loading frequency is limited only by the storage target.

Stream Transformer has three variants: [Transformer Kinesis](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/), [Transformer Pubsub](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/) and [Transformer Kafka](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/). They are different variants for AWS, GCP and Azure.

---

# Transformer Kafka configuration reference
> Configure Transformer Kafka with stream settings, Azure Blob Storage output, windowing, and monitoring for Azure real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/configuration-reference/

The configuration reference in this page is written for Transformer Kafka `6.6.0`

An example of the minimal required config for the Transformer Kafka can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

| Parameter                                            | Description                                                                                                                                                               |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                                    | Name of the Kafka topic to read from                                                                                                                                      |
| `input.bootstrapServers`                             | A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                                                             |
| `input.consumerConf`                                 | Optional. Kafka consumer configuration. See <https://kafka.apache.org/documentation/#consumerconfigs> for all properties                                                  |
| `output.path`                                        | Azure Blob Storage path to transformer output                                                                                                                             |
| `output.compression`                                 | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                 |
| `output.bad.type`                                    | Optional. Either `kafka` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`. |
| `output.bad.topicName`                               | Required if output type is `kafka`. Name of the Kafka topic that will receive the bad data.                                                                               |
| `output.bad.bootstrapServers`                        | Required if output type is `kafka`. A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                         |
| `output.producerConf`                                | Optional. Kafka producer configuration. See <https://kafka.apache.org/documentation/#producerconfigs> for all properties                                                  |
| `queue.topicName`                                    | Name of the Kafka topic used to communicate with Loader                                                                                                                   |
| `queue.bootstrapServers`                             | A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                                                             |
| `queue.producerConf`                                 | Optional. Kafka producer configuration. See <https://kafka.apache.org/documentation/#producerconfigs> for all properties                                                  |
| `monitoring.metrics.*`                               | Send metrics to a StatsD server or stdout.                                                                                                                                |
| `monitoring.metrics.statsd.*`                        | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                             |
| `monitoring.metrics.statsd.hostname`                 | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                        |
| `monitoring.metrics.statsd.port`                     | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                 |
| `monitoring.metrics.statsd.tags`                     | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                              |
| `monitoring.metrics.statsd.prefix`                   | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                             |
| `monitoring.metrics.stdout.*`                        | Optional. For sending metrics to stdout.                                                                                                                                  |
| `monitoring.metrics.stdout.prefix`                   | Optional. Overrides the default metric prefix.                                                                                                                            |
| `telemetry.disable`                                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                 |
| `telemetry.userProvidedId`                           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                       |
| `monitoring.sentry.dsn`                              | Optional. For tracking runtime exceptions.                                                                                                                                |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0) | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                           |
| `validations.*`                                      | Optional. Criteria to validate events against                                                                                                                             |
| `validations.minimumTimestamp`                       | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.        |
| `featureFlags.*`                                     | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                               |
| `featureFlags.legacyMessageFormat`                   | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                      |
| `featureFlags.truncateAtomicFields` (since 5.4.0)    | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.              |

---

# Transformer Kafka for Azure streams
> Stream transformer for Azure that reads enriched events from Kafka and writes transformed data to Azure Blob Storage in real-time.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/

> **Info:** This application is available since v5.7.0.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/transformer-kafka:6.6.0`

## Configuring `snowplow-transformer-kafka`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Transformer Kafka can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Transformer Kafka

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/transformer-kafka:6.6.0 \

--iglu-config $RESOLVER_BASE64 \

--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Transformer Kafka (since version 5.7.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Transformer Kinesis configuration reference
> Configure Transformer Kinesis with stream settings, S3 output, windowing, and monitoring for AWS real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/configuration-reference/

The configuration reference in this page is written for Transformer Kinesis `6.6.0`

An example of the minimal required config for the Transformer Kinesis can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

| Parameter                                                    | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.appName`                                              | Optional. KCL app name. The default is `snowplow-rdb-transformer`                                                                                                                                                                                                                                                                                                                                             |
| `input.streamName`                                           | Required for `kinesis`. Enriched Kinesis stream name.                                                                                                                                                                                                                                                                                                                                                         |
| `input.region`                                               | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `input.position`                                             | Optional. Kinesis position: `LATEST` or `TRIM_HORIZON`. The default is `LATEST`.                                                                                                                                                                                                                                                                                                                              |
| `windowing`                                                  | Optional. Frequency to emit shredding complete message. The default is `10 minutes`. Maximum allowed value is `60 minutes`                                                                                                                                                                                                                                                                                    |
| `output.path`                                                | Required. S3 URI of the transformed output.                                                                                                                                                                                                                                                                                                                                                                   |
| `output.compression`                                         | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                                                                                                                                     |
| `output.region`                                              | AWS region of the S3 bucket. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                          |
| `output.maxRecordsPerFile` (since 5.4.0)                     | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.type` (since 5.4.0)                              | Optional. Either `kinesis` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                                                                                                                                   |
| `output.bad.streamName` (since 5.4.0)                        | Required if output type is `kinesis`. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                                                                                                                 |
| `output.bad.region` (since 5.4.0)                            | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `output.bad.recordLimit` (since 5.4.0)                       | Optional, default = 500. Limits the number of events in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                                  |
| `output.bad.byteLimit` (since 5.4.0)                         | Optional, default = 5242880. Limits the number of bytes in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                               |
| `output.bad.backoffPolicy.minBackoff` (since 5.4.0)          | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                     |
| `output.bad.backoffPolicy.maxBackoff` (since 5.4.0)          | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                           |
| `output.bad.backoffPolicy.maxRetries` (since 5.4.0)          | Optional, default = 10. Maximum number of retries for internal Kinesis errors.                                                                                                                                                                                                                                                                                                                                |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 5.4.0) | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails in case of throughput exceeded.                                                                                                                                                                                                                                                                           |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 5.4.0) | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails in case of throughput exceeded. Writing is retried forever.                                                                                                                                                                                                                                                     |
| `queue.type`                                                 | Required. Type of the message queue. Can be either `sqs` or `sns`.                                                                                                                                                                                                                                                                                                                                            |
| `queue.queueName`                                            | Required if queue type is `sqs`. Name of the SQS queue. SQS queue needs to be FIFO.                                                                                                                                                                                                                                                                                                                           |
| `queue.topicArn`                                             | Required if queue type is `sns`. ARN of the SNS topic.                                                                                                                                                                                                                                                                                                                                                        |
| `queue.region`                                               | AWS region of the SQS queue or SNS topic. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                             |
| `formats.*`                                                  | Schema-specific format settings.                                                                                                                                                                                                                                                                                                                                                                              |
| `formats.transformationType`                                 | Required. Type of transformation, either `shred` or `widerow`. See [Shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) and [Wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format).                                                                             |
| `formats.fileFormat`                                         | Optional. The default is `JSON`. Output file format produced when transformation is `widerow`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                                                                                                    |
| `formats.default`                                            | Optional. The default is `TSV`. Data format produced by default when transformation is `shred`. Either `TSV` or `JSON`. `TSV` is recommended as it enables table autocreation, but requires an Iglu Server to be available with known schemas (including Snowplow schemas). `JSON` does not require an Iglu Server, but requires Redshift JSONPaths to be configured and does not support table autocreation. |
| `formats.tsv`                                                | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `JSON` this list of schemas will still be shredded into `TSV`.                                                                                                                                                                                                                                    |
| `formats.json`                                               | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `TSV` this list of schemas will still be shredded into `JSON`.                                                                                                                                                                                                                                    |
| `formats.skip`                                               | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. Schemas for which loading can be skipped.                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.*`                                       | Send metrics to a StatsD server or stdout.                                                                                                                                                                                                                                                                                                                                                                    |
| `monitoring.metrics.statsd.*`                                | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.hostname`                         | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                                                                                                                                                                                                            |
| `monitoring.metrics.statsd.port`                             | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                                                                                                                                                                                                                     |
| `monitoring.metrics.statsd.tags`                             | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.prefix`                           | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.stdout.*`                                | Optional. For sending metrics to stdout.                                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.stdout.prefix`                           | Optional. Overrides the default metric prefix.                                                                                                                                                                                                                                                                                                                                                                |
| `telemetry.disable`                                          | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                     |
| `telemetry.userProvidedId`                                   | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.dsn`                                      | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                    |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0)         | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                                                                                                                                               |
| `validations.*`                                              | Optional. Criteria to validate events against                                                                                                                                                                                                                                                                                                                                                                 |
| `validations.minimumTimestamp`                               | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                                                                                                                                            |
| `featureFlags.*`                                             | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                                                                                                                                                   |
| `featureFlags.legacyMessageFormat`                           | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                                                                                                                                          |
| `featureFlags.truncateAtomicFields` (since 5.4.0)            | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                                                                                                                                                  |

---

# Transformer Kinesis for AWS streams
> Stream transformer for AWS that reads enriched events from Kinesis and writes transformed data to S3 in real-time.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/transformer-kinesis:6.6.0`

## Configuring `snowplow-transformer-kinesis`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Transformer Kinesis can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Transformer Kinesis

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/transformer-kinesis:6.6.0 \

--iglu-config $RESOLVER_BASE64 \

--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Transformer Kinesis (since version 4.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Transformer Pub/Sub configuration reference
> Configure Transformer Pub/Sub with stream settings, GCS output, windowing, and monitoring for GCP real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/configuration-reference/

The configuration reference in this page is written for Transformer Pubsub `6.6.0`

An example of the minimal required config for the Transformer Pubsub can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

| Parameter                                            | Description                                                                                                                                                                                                                                                                               |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                                 | Name of the Pubsub subscription with the enriched events                                                                                                                                                                                                                                  |
| `input.parallelPullCount`                            | Optional. Default value 1. Number of threads used internally by permutive library to handle incoming messages. These threads do very little "work" apart from writing the message to a concurrent Queue.                                                                                  |
| `input.bufferSize`                                   | Optional. Default value 500. The max size of the buffer queue used between fs2-pubsub and java-pubsub libraries.                                                                                                                                                                          |
| `input.maxAckExtensionPeriod`                        | Optional. Default value '1 hour'. The maximum period a message ack deadline will be extended.                                                                                                                                                                                             |
| `output.path`                                        | Required. GCS URI of the transformed output. It needs to have `gs://` URI scheme                                                                                                                                                                                                          |
| `output.compression`                                 | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                 |
| `output.bufferSize`                                  | Optional. Default value 4096. During the window period, processed items are stored in a buffer. This value determines the size of this buffer. When its limit is reached, buffer content is flushed to blob storage.                                                                      |
| `output.maxRecordsPerFile` (since 5.4.0)             | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                    |
| `output.bad.type` (since 5.4.0)                      | Optional. Either `pubsub` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                |
| `output.bad.topic` (since 5.4.0)                     | Required if output type is `pubsub`. Name of the PubSub topic that will receive the bad data.                                                                                                                                                                                             |
| `output.bad.batchSize` (since 5.4.0)                 | Optional. Default = 1000, max = 1000. Maximum number of messages sent to PubSub within a batch. When the buffer reaches this number of messages they are sent.                                                                                                                            |
| `output.bad.requestByteThreshold` (since 5.4.0)      | Optional. Default = 8000000, max = 10MB. Maximum number of bytes sent to PubSub within a batch. When the buffer reaches this size messages are sent.                                                                                                                                      |
| `output.bad.delayThreshold` (since 5.4.0)            | Optional. Default = 200 milliseconds. Delay threshold to use for PubSub batching. After this amount of time has elapsed, before `batchSize` and `requestByteThreshold` have been reached, messages from the buffer will be sent.                                                          |
| `queue.topic`                                        | Name of the Pubsub topic used to communicate with Loader                                                                                                                                                                                                                                  |
| `formats.fileFormat`                                 | Optional. The default option at the moment is `JSON`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                         |
| `windowing`                                          | Optional. Frequency to emit shredding complete message. The default is `5 minutes`. Note that there is a problem with acking messages when window period is greater than 10 minute in transformer-pubsub. Therefore, it is advisable to make window period equal or less than 10 minutes. |
| `monitoring.metrics.*`                               | Send metrics to a StatsD server or stdout.                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.*`                        | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                                                                                                                                             |
| `monitoring.metrics.statsd.hostname`                 | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                                                                                        |
| `monitoring.metrics.statsd.port`                     | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.tags`                     | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                                                                                              |
| `monitoring.metrics.statsd.prefix`                   | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                                                                                                                                             |
| `monitoring.metrics.stdout.*`                        | Optional. For sending metrics to stdout.                                                                                                                                                                                                                                                  |
| `monitoring.metrics.stdout.prefix`                   | Optional. Overrides the default metric prefix.                                                                                                                                                                                                                                            |
| `telemetry.disable`                                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                 |
| `telemetry.userProvidedId`                           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                       |
| `monitoring.sentry.dsn`                              | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0) | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                           |
| `validations.*`                                      | Optional. Criteria to validate events against                                                                                                                                                                                                                                             |
| `validations.minimumTimestamp`                       | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                        |
| `featureFlags.*`                                     | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                               |
| `featureFlags.legacyMessageFormat`                   | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                      |
| `featureFlags.truncateAtomicFields` (since 5.4.0)    | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                              |

---

# Transformer Pub/Sub for GCP streams
> Stream transformer for GCP that reads enriched events from Pub/Sub and writes transformed data to GCS in real-time.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/

> **Info:** This application is available since v5.0.0.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/transformer-pubsub:6.6.0`

## Configuring `snowplow-transformer-pubsub`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Transformer Pubsub can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/#iglu-central) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Transformer Pubsub

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/transformer-pubsub:6.6.0 \

--iglu-config $RESOLVER_BASE64 \

--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Transformer PubSub (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# RDB Loader 1.0.0 upgrade guide
> Upgrade RDB Loader to 1.0.0 with Stream Shredder, unified output partitioning, and new configuration schema for batch processing.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/1-0-0-upgrade-guide/

This is a release adding a new experimental Stream Shredder asset and improving independent Loader architecture, introduced in R35.

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/1.0.0).

This is the first release in 1.x branch and no breaking changes will be introduced until 2.x release. If you're upgrading from R34 or earlier it's strictly recommended to follow [R35 Upgrade Guide](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r35-upgrade-guide/) first.

## Assets

RDB Loader, RDB Shredder and Stream RDB Shredder all have 1.o.0 version, despite last one being an experimental asset. RDB Shredder is published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-1.0.1.jar`

RDB Loader and RDB Stream Shredder distributed as Docker images, published on DockerHub:

- `snowplow/snowplow-rdb-loader:1.0.1`
- `snowplow/snowplow-rdb-stream-shredder:1.0.1`

## Configuration changes

All configuration changes are scoped to `shredder` property.

Since we added another type of a shredder, one has to specify the type explicitly:

```json


"shredder": {

  "type" : "batch",                                  # Was not necessary in R35

  "input": "s3://snowplow-enriched-archive/path/",   # Remains the same

  "output": ...                                      # Explained below

}
```

\_

The major API change in 1.0.0 is the new partitioning scheme unifying `good` and `bad` output. Whereas previously it was necessary to specify `output` and `outputBad`, now there's only `path` in `shredder.output` object:

\_

_`"output": { # Was a string in R35 "path": "s3://snowplow-shredded-archive/", # Path to shredded output "compression": "GZIP" # Output compression, GZIP or NONE`_ `}`

In Dataflow Runner playbook you have to specify new Main classpath for RDB Shredder:

```text
"--class", "com.snowplowanalytics.snowplow.rdbloader.shredder.batch.Main"
```

## Manifest

The new manifest table has the same name as previous one - `manifest`. In order to avoid a clash, RDB Loader 1.0.0 checks existence of the table every time it starts and if table exists checks if it's new or old one. If table exists and it's legacy - it will be renamed into `manifest_legacy` and can be removed manually later, new table will be created automatically. If table doesn't exist - it will be created.

No user actions necessary here.

## Stream Shredder

You only need to choose one Shredder: batch or stream. **For production environment we recommend using Batch Shredder.**

Stream Shredder is configured within same configuration file as RDB Loader and RDB Batch Shredder, but using following properties:

```json
  "shredder": {

    # A batch loader would fail, if stream type encountered

    "type" : "stream",

    # Input stream information

    "input": {

      # file is another option, but used for debugging only

      "type": "kinesis",

      # KCL app name - a DynamoDB table will be created with the same name

      "appName": "acme-rdb-shredder",

      # Kinesis Stream name, must exist

      "streamName": "enriched-events",

      # Kinesis region

      "region": "us-east-1",

      # Kinesis position: LATEST or TRIM_HORIZON

      "position": "LATEST"

    },



    # A frequency to emit loading finished message - 5,10,15,20,30,60 etc minutes, this is what controls how often your data will be loaded

    "windowing": "10 minutes",



    # Path to shredded archive, same as for batch

    "output": {

      # Path to shredded output

      "path": "s3://bucket/good/",

      # Shredder output compression, GZIP or NONE

      "compression": "GZIP"

    }

}
```

## Directory structure

There is a major change in shredder output directory structure, on top of what has changed in R35.

If you're using a 3rd-party query engine such as Amazon Athena to query shredded data, the new partitioning can break the schema. And thus it's recommended to create a new root for shredded data.

Structure of the typical shredded folder now looks like following:

```text
run=2021-03-29-15-40-30/

    shredding_complete.json

    output=good/

            vendor=com.snowplowanalytics.snowplow/

                name=atomic/

                    format=tsv/

                        model=1/

            vendor=com.acme/

                name=link_click/

                    format=json/

                        model=1/

    output=bad/

            vendor=com.snowplowanalytics.snowplow/

                name=loader_parsing_error/

                    format=json/

                        model=1/
```

---

# RDB Loader 1.2.0 upgrade guide
> Upgrade RDB Loader to 1.2.0 with webhook monitoring, folder monitoring, and enhanced observability for Redshift loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/1-2-0-upgrade-guide/

RDB Loader 1.2.0 brings many improvements for monitoring subsystem. If you're not interested in new features - you can just bump versions. If you need webhook monitoring - read below instructions on how to enable it.

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/1.2.0).

## Assets

RDB Shredder is published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-1.2.0.jar`

RDB Loader and RDB Stream Shredder distributed as Docker images, published on DockerHub:

- `snowplow/snowplow-rdb-loader:1.2.0`
- `snowplow/snowplow-rdb-stream-shredder:1.2.0`

## Enabling Webhook monitoring

All configuration changes are scoped to `monitoring` property.

```json
"monitoring": {

  "webhook": {

    "endpoint": "https://webhooks.acme.com/rdb-loader",

    "tags": {            # Custom set of tags

      "host": $HOST,     # Environment variables are supported

      "pipeline": "production"

    }

  }

}
```

It's up to you to setup a preferable webhook backend. It can be a Snowplow Iglu webhook or custom monitoring system.

## Enabling folder monitoring

All configuration changes are scoped to `monitoring` property.

```json
"monitoring": {

  "folders": {

    "staging": "s3://snowplow-acme-com/logging/",    # This path will contain temporary files

                                                     # Redshift role must have an access for this folder

    "period": "2 hours"                              # How often the check should be performed

  }

}
```

It's up to you to setup a preferable webhook backend. It can be a Snowplow Iglu webhook or custom monitoring system.

---

# RDB Loader 2.0.0 upgrade guide
> Upgrade RDB Loader to 2.0.0 with separate configs for Loader and Shredder, SNS messaging, and split configuration architecture.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/2-0-0-upgrade-guide/

RDB Loader 2.0.0 brings ability to send shredding complete messages from Shredder to SNS topic and splits configs of RDB Loader and RDB Shredder. From now on, Loader and Shredder will not use same config.

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/2.0.0).

## Assets

RDB Shredder is published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-2.0.0.jar`

RDB Loader and RDB Stream Shredder distributed as Docker images, published on DockerHub:

- `snowplow/snowplow-rdb-loader:`2.0.0
- `snowplow/snowplow-rdb-stream-shredder:`2.0.0

## Sending shredding complete messages from Shredder to SNS

Shredding complete message can be sent to SNS topic with following queue configuration:

```json
"queue": {

  "type": "sns",

  "topicArn": "arn:aws:sns:eu-central-1:123456789:test-sns-topic",

  "region": "eu-central-1"

}
```

## New separate configs

RDB Loader and RDB Shredder were using the same config HOCON until version 2.0.0. Starting from 2.0.0, they will use separate configs. Reference docs for new configs can be found on the following pages:

[RDB Loader configuration](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/previous-versions/snowplow-rdb-loader/configuration-reference/)

[RDB Shredder configuration](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/previous-versions/snowplow-rdb-loader/rdb-shredder-configuration-reference/)

---

# RDB Loader 6.0.0 upgrade guide
> Upgrade RDB Loader to 6.0.0 with recovery tables, schema merging, and improved schema evolution for Redshift and other warehouses.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/6-0-0-upgrade-guide/

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

## \[Redshift-only] New migration mechanism & recovery tables

### What is schema evolution?

One of Snowplow’s key features is the ability to [define custom schemas and validate events against them](/docs/fundamentals/schemas/). Over time, users often evolve the schemas, e.g. by adding new fields or changing existing fields. To accommodate these changes, RDB loader automatically adjusts the database tables in the warehouse accordingly.

There are two main types of schema changes:

**Breaking**: The schema version has to be changed in a model way (`1-2-3` → `2-0-0`). In Redshift, each model schema version has its own table (`..._1`, `..._2`, etc, for example: `com_snowplowanalytics_snowplow_ad_click_1`).

**Non-breaking**: The schema version can be changed in a addition way (`1-2-3` → `1-3-0` or `1-2-3` → `1-2-4`). Data is stored in the same database table.

### How it used to work

In the past, the transformer would fetch all schemas for an entity (which conforms to the `vendor/name/model-*-*` criterion of the entity) from Iglu Server, merge them, extract properties from JSON, stringify using tab char as delimeter and put null char in case the property is missing. Then the loader would adjust the database table and load the file.

This logic relied on two assumptions:

1. **Old events compatible with new schemas.** Events with older schema versions, e.g. `1-0-0` and `1-0-1`, had to be valid against the newer ones, e.g. `1-0-2`. Those that were not valid would result in failed events.

2. **Old columns compatible with new schemas.** The corresponding Redshift table had to be migrated correctly from one version to another. Changes, such as altering the type of a field from `integer` to `string`, would fail. Loading would break with SQL errors and the whole batch would be stuck and hard to recover.

These assumptions were not always clear to the users, making the transformer and loader error-prone.

### What happens now?

Transformer and loader are now more robust, and the data is easy to recover if the schema was not evolved correctly.

First, we support schema evolution that’s not strictly backwards compatible (although we still recommend against it since it can confuse downstream consumers of the data). This is done by _merging_ multiple schemas so that both old and new events can coexist. For example, suppose we have these two schemas:

```json
{

   // 1-0-0

   "properties": {

      "a": {"type": "integer"}

   }

}
```

```json
{

   // 1-0-1

   "properties": {

      "b": {"type": "integer"}

   }

}
```

These would be merged into the following:

```json
{

   // merged

   "properties": {

      "a": {"type": "integer"},

      "b": {"type": "integer"}

   }

}
```

Second, the loader does not fail when it can’t modify the database table to store both old and new events. (As a reminder, an example would be changing the type of a field from `integer` to `string`.) Instead, it creates a table for the new data as an exception. The users can then run SQL statements to resolve this situation as they see fit. For instance, consider these two schemas:

```json
{

   // 1-0-0

   "properties": {

      "a": {"type": "integer"}

   }

}
```

```json
{

   // 1-0-1

   "properties": {

      "a": {"type": "string"}

   }

}
```

Because `1-0-1` events cannot be loaded into the same table with `1-0-0`, the data would be put in a separate table, e.g. `com_snowplowanalytics_ad_click_1_0_1_recovered_9999999`, where:

- `1_0_1` is the version of the offending schema;
- `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one).

If you create a new schema `1-0-2` that reverts the offending changes and is again compatible with `1-0-0`, the data for events with that schema will be written to the original table as expected.

### Identifying schemas that need patching

After upgrading RDB Loader, you might find out that events or entities with some of the old schemas land in recovery tables.

To avoid this, the offending older Iglu schemas must be patched to align with the latest.

You can use the latest version of `igluctl` to do this.

To illustrate, let's say we have schema versions `1-0-0` and `1-0-1` that differ in one field only:

- version `1-0-0`: `{ "type": "integer", "maximum": 100 }` - translates to `SMALLINT`. All is good.
- version `1-0-1`: `{ "type": "integer", "maximum": 1000000 }` - breaking change as it translates to `INT` and the loader can't migrate the column.

Since version `1-0-1` had a breaking change, loading must have broken with an older version of RDB Loader. To fix that, the user must have updated the warehouse manually (changing the column type to `INT`). After this manual intervention, events with the `1-0-1` schema would have been loaded successfully with older versions of RDB Loader. However, after an upgrade to RDB Loader 6.0.0, events with the `1-0-1` schema will start to land in the recovery table.

Let's see how we can use `igluctl` to solve this problem.

1. Run the `igluctl static generate` command. If a recovery table is to be created, it will show up as a warning message. Example:

```bash
mkdir <schemas_folder> <sql_folder>

igluctl static pull <schemas_folder> <iglu_url> <iglu_key>

igluctl static generate <schemas_folder> <sql_folder>

# ...

# iglu:com.test/test/jsonschema/1-0-1 has a breaking change Incompatible types in column example_field old RedshiftSmallInt new RedshiftInteger

# ...
```

2. Run the `igluctl table-check` command to check if table structure is in line with the latest schema version that doesn't contain any breaking changes. With the example schemas above, this would be `1-0-0` because `1-0-1` contains a breaking change. Example:

```bash
igluctl table-check \

        --server <iglu_url> \

        --apikey <iglu_key> \

        --host <redshift_host> \

        --port <redshift_port> \

        --username <username> \

        --password <password> \

        --dbname  <database> \

        --dbschema <schema>

# ...

# * Column doesn't match, expected: 'example_field SMALLINT', actual: 'example_field INT'

# ...
```

We got the `Column doesn't match` output with the above example because the table column had been migrated manually from `SMALLINT` to `INT`. Since the latest schema version that doesn't contain any breaking change is `1-0-0`, `table-check` command expects to see `SMALLINT` in the table therefore it gives the `Column doesn't match` output.

In order to solve this problem, we should patch `1-0-0` with `{ "type": "integer", "maximum": 1000000 }`. In this case, there won't be any breaking change between versions `1-0-0` and `1-0-1`. RDB Loader 6.0.0+ will then successfully load events into the intended table as expected.

After identifying all the offending schemas, you should patch them to reflect the changes in the warehouse.

Schema casting rules could be found [here](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=redshift#types).

#### `$.featureFlags.disableRecovery` configuration

If you have older schemas with breaking changes and don’t want the loader to apply the new logic to them, you can use `$.featureFlags.disableRecovery` configuration. For the provided schema criterions only, RDB Loader will neither migrate the corresponding shredded table nor create recovery tables for breaking schema versions. Loader will attempt to load to the corresponding shredded table without migrating.

You can set it as follows:

```json
{

  ...

  "featureFlags": {

    "disableRecovery": [

      "iglu:com.example/myschema1/jsonschema/1-*-*",

      "iglu:com.example/myschema2/jsonschema/1-*-*"

    ]

  }

}
```

---

# Upgrade guides for the Snowplow RDB Loader
> Step-by-step upgrade guides for RDB Loader with breaking changes, schema migrations, and compatibility notes for Redshift, Snowflake, and Databricks.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/

This section contains information to help you upgrade to newer versions of Snowplow RDB Loader.

---

# RDB Loader R32 upgrade guide
> Upgrade RDB Loader to R32 with EMR 5.19.0, automigrations, and updated shredder and loader versions for Redshift.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r32-upgrade-guide/

We recommend to go through the upgrade routine in several independent steps. After every step you should have a working pipeline. If something is not working or Shredder produces unexpected failed events - please let us know.

## Updating assets

1. Upgrade EmrEtlRunner to R116 or higher

2. In your `redshift_config.json`

   1. Update SchemaVer to `4-0-0`
   2. Add `"blacklistTabular": null` field into `data` payload

3. Update your `config.yml` file

```yaml
aws:

  emr:

    ami_version: 5.19.0     # was 5.9.0; Required by RDB Shredder

storage:

  versions:

    rdb_loader: 0.17.0      # was 0.16.0

    rdb_shredder: 0.16.0    # was 0.15.0
```

At this point, your pipeline should be running with new assets as it was before, without automigrations and generated TSV. We recommend to test this setup and monitor shredded failed events for one or two runs before proceeding to enabling automigrations.

## Iglu Server

Automigrations work only with Iglu Server 0.6.0. This component provides information about how columns should be ordered across different ADDITIONs and REVISIOINs. If you still don't have Iglu Server 0.6.0 we recommend to [set it up](https://github.com/snowplow/iglu/wiki/Setting-up-an-Iglu-Server).

You still can use static registries as a backup, they will continue to work for validatioin purposes, but won't work for TSV shredding. Snowplow does not provide a public Iglu Server hosting Iglu Central schemas, so we recommend you to mirror Iglu Central with your own Iglu Server:

```bash
$ git clone https://github.com/snowplow/iglu-central.git

$ igluctl static push iglu-central/schemas $YOUR_SERVER_URL $YOU_API_KEY

$ igluctl static push com.acme-iglu-registry/schemas $YOUR_SERVER_URL $YOU_API_KEY
```

After setting up the Iglu Server, don't forget to add it to your resolver config.

## Tabular blacklisting

New RDB Shredder is still able to produce legacy JSON files. But automigrations can be applied only to tables where data is prepared as TSV. If you setup a new pipeline, you can generate only TSVs abandoning legacy DDLs (except `atomic.events` and `atomic.manifest`) and JSONPaths altogether. However, if you already have tables deployed which DDLs were generated manually or via old igluctl you will likely need to apply so called _tabular blacklisting_ to these tables. It means that Shredder will keep producing data with these schemas as JSONs and Loader won't be able to apply migrations to it. This is necessary because manually generated DDLs are not guaranteed to have predictable column order and the only way to map JSON values to respective columns is JSONPaths files.

[igluctl version 0.7.0](https://github.com/snowplow/igluctl/releases/tag/0.7.0) provides `rdbms table-check` subcommand that get schemas from Iglu Server, figures out what DDL the Loader would generate, then connects to Redshift and compares those DDLs with actual state of the table. Every table that have an incompatible order will have to be "blacklisted" in Redshift storage target config (`redshift_config.json`).

Here's an example of a black list containing several schemas from Iglu Central:

```json
  "blacklistTabular": [

    "iglu:org.w3/PerformanceTiming/jsonschema/1-*-*",



    "iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/application_error/jsonschema/1-*-*",



    "iglu:com.mandrill/recipient_unsubscribed/jsonschema/1-*-*",

    "iglu:com.mandrill/message_soft_bounced/jsonschema/1-*-*",

    "iglu:com.mandrill/message_sent/jsonschema/1-*-*",

    "iglu:com.mandrill/message_opened/jsonschema/1-*-*",

    "iglu:com.mandrill/message_marked_as_spam/jsonschema/1-*-*",

    "iglu:com.mandrill/message_delayed/jsonschema/1-*-*",

    "iglu:com.mandrill/message_clicked/jsonschema/1-*-*",

    "iglu:com.mandrill/message_bounced/jsonschema/1-*-*"

  ]
```

As you can see, schemas specified in schema criterion format (with wildcards everywhere except MODEL).

## Conclusion

At this point if you track an event with new schema and this schema resides on an Iglu Server - RDB Shredder will produce TSV data for it and RDB Loader will automatically create a new table. Same with ADDITION and REVISION migrations - they're handled by RDB Loader automatically.

---

# RDB Loader R33/R34 upgrade guide
> Upgrade RDB Loader to R33/R34 with Spark 3, EMR 6.1.0, and bugfixes for long text properties in Redshift.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r33-upgrade-guide/

R34 is a release with bugfixes and performance improvements. R33 was almost identical reelase with major bug preventing some long text properties from loading.

## Updating assets

1. Upgrade EmrEtlRunner to 1.0.4 or higher
2. Your `redshift_config.json` should have 4-0-0 version
3. Update your `config.yml` file

```yaml
aws:

  emr:

    ami_version: 6.1.0     # was 5.19.0; Required by Spark 3

storage:

  versions:

    rdb_loader: 0.18.2      # was 0.17.0

    rdb_shredder: 0.18.2    # was 0.16.0
```

---

# RDB Loader R35 upgrade guide
> Upgrade RDB Loader to R35 with independent Loader architecture, SQS messaging, and removal of EmrEtlRunner dependency.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r35-upgrade-guide/

R35 is a release with major changes in pipeline architecture:

- No dependency on EmrEtlRunner (neither Shredder nor Loader can be launched using EmrEtlRunner, marking deprecation of EmrEtlRunner)
- Loader is not an EMR step anymore
- Major changes in directory structure
- New dependency on SQS

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/r35).

This is the last release in 0.x branch and breaking changes still might be introduced in 1.0.0 release.

## Assets

Both RDB Shredder and Loader have 0.19.0 version. Both published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-0.19.0.jar`

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-loader/snowplow-rdb-loader-0.19.0.jar`

For RDB Loader however it is recommended to use the docker image, published on DockerHub: `snowplow/snowplow-rdb-loader:0.19.0`

## New architecture

Previous workflow was orchestrated by EmrEtlRunner, along with multiple S3DistCp steps, recovery scenarios and dedicated RDB Loader step. RDB Loader was finding out what data needs to be loaded by scanning S3.

In the new architecture there are two EMR steps:

1. S3DistCp, copying enriched data sunk by [S3 Loader](/docs/api-reference/loaders-storage-targets/s3-loader/), from S3 sink bucket (same as "enriched stream bucket") into _enriched data lake_ (aka shredder input, similar as previously known "enriched archive")
2. RDB Shredder, picking up all _unprocessed_ folders in enriched data lake, shredding data there and writing it into _shredded data lake_ (previously known as "shredded archive")

RDB Loader is a stand-alone long-running app, launched either on EC2 box or Fargate cluster. The loading gets triggered by an SQS message, sent by Shredder after it finished processing a new batch.

RDB Shredder decides that folder is unprocessed by:

1. Comparing folder names in _enriched data lake_ and in _shredded data lake_. Every folder that is **in** enriched, but **not in** shredded will be considered unprocessed
2. ...except folders that don't have `shredding_complete.json` file in their root. This file is written at the end of the job and indicates that job has completed successfully. Absence of this file means that shred job has been aborted.

If you're upgrading from R34 or earlier it is strictly recommended to pick new paths for enriched and shredded archives in order to avoid double-loading OR make sure that there's a strict 1:1 correspondence between content of enriched and shredded archive.

We recommend to use either [Dataflow Runner](/docs/api-reference/dataflow-runner/) or boto3 script to launch scheduled S3DistCp and Shredder jobs. Here's an example of a Dataflow Runner playbook:

```json
{

  "schema": "iglu:com.snowplowanalytics.dataflowrunner/PlaybookConfig/avro/1-0-1",

  "data": {

    "region": "eu-central-1",

    "credentials": {

      "accessKeyId": "env",

      "secretAccessKey": "env"

    },

    "steps": [

      {

        "type": "CUSTOM_JAR",

        "name": "S3DistCp enriched data archiving",

        "actionOnFailure": "CANCEL_AND_WAIT",

        "jar": "/usr/share/aws/emr/s3-dist-cp/lib/s3-dist-cp.jar",

        "arguments": [

            "--src", "s3://com-acme/enriched/sink/",

            "--dest", "s3://com-acme/enriched/archive/run={{nowWithFormat "2006-01-02-15-04-05"}}/",

            "--s3Endpoint", "s3-eu-central-1.amazonaws.com",

            "--srcPattern", ".*",

            "--outputCodec", "gz",

            "--deleteOnSuccess"

        ]

      },



      {

        "type": "CUSTOM_JAR",

        "name": "RDB Shredder",

        "actionOnFailure": "CANCEL_AND_WAIT",

        "jar": "command-runner.jar",

        "arguments": [

            "spark-submit",

            "--class", "com.snowplowanalytics.snowplow.shredder.Main",

            "--master", "yarn",

            "--deploy-mode", "cluster",

            "s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-0.19.0.jar",

            "--iglu-config", "{{base64File "/home/snowplow/configs/snowplow/iglu_resolver.json"}}",

            "--config", "{{base64File "/home/snowplow/configs/snowplow/config.hocon"}}"

        ]

      }

    ],

    "tags": [ ]

  }

}
```

We recommend to launch RDB Loader as long-running docker image.

## New configuration file

Common configuration file, previously known as `config.yml` and target JSON configuration file, previously known as `redshift.json` have been replaced by a [single HOCON file](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/config.hocon.sample).

Here's an example:

```json
{

  # Human-readable identificator, can be random

  "name": "Acme Redshift",

  # Machine-readable unique identificator, must be UUID

  "id": "123e4567-e89b-12d3-a456-426655440000",



  # Data Lake (S3) region

  "region": "us-east-1",

  # SQS topic name used by Shredder and Loader to communicate

  "messageQueue": "messages.fifo",



  # Shredder-specific configs

  "shredder": {

    # Path to enriched archive (must be populated separately with run=YYYY-MM-DD-hh-mm-ss directories)

    "input": "s3://com-acme/ernched/archive/",

    # Path to shredded output

    "output": "s3://com-acme/shredded/good/",

    # Path to data failed being processed

    "outputBad": "s3://com-acje/shredded/bad/",

    # Shredder output compression, GZIP or NONE

    "compression": "GZIP"

  },



  # Optional. S3 path that holds JSONPaths

  "jsonpaths": "s3://bucket/jsonpaths/",



  # Schema-specific format settings (recommended to leave all three groups empty and use TSV as default)

  # To make it compatible with R34, leave default = TSV and populate json array with things from blacklistTabular

  "formats": {

    # Format used by default (TSV or JSON)

    "default": "TSV",

    # Schemas to be shredded as JSONs, corresponding JSONPath files must be present. Automigrations will be disabled

    "json": [ ],

    # Schemas to be shredded as TSVs, presence of the schema on Iglu Server is necessary. Automigartions enabled

    "tsv": [ ],

    # Schemas that won't be loaded

    "skip": [ ]

  },



  # Warehouse connection details, identical to storage target config

  "storage": {

    # Database, redshift is the only acceptable option

    "type": "redshift",

    # Redshift hostname

    "host": "redshift.amazon.com",

    # Database name

    "database": "snowplow",

    # Database port

    "port": 5439,

    # AWS Role ARN allowing Redshift to load data from S3

    "roleArn": "arn:aws:iam::123456789012:role/RedshiftLoadRole",

    # DB schema name

    "schema": "atomic",

    # DB user with permissions to load data

    "username": "storage-loader",

    # DB password

    "password": "secret",

    # Custom JDBC configuration

    "jdbc": {"ssl": true},

    # MAXERROR, amount of acceptable loading errors

    "maxError": 10,

    "compRows": 100000

  },



  # Additional steps. analyze, vacuum and transit_load are valid values

  "steps": ["analyze"],



  # Observability and logging opitons

  "monitoring": {

    # Snowplow tracking (optional)

    "snowplow": null,

    # Sentry (optional)

    "sentry": null

  }

}
```

If you need to use cross-batch deduplication - the file format remains the same for DynamoDB config.

CLI arguments also have changed. Both applications now accept only `--iglu-config` with base64-encoded string representing [Iglu Resolver JSON](/docs/api-reference/iglu/iglu-resolver/) and `--config` with base64-encoded above HOCON. Loader also accepts `--dry-run` flag.

## SQS

SQS serves as message bus between Shredder and Loader. Loader expects to find there self-describing messages with instructions on what to load. The queue must be FIFO.

## Directory structure

There are several major changes in shredder output directory structure:

1. Elements of the paths have changed from Iglu-compatible to shredder-specific, e.g. `format` now can be either `json` or `tsv` (and not `jsonschema` as before) and instead of `version` (that could have been either `1-0-0` or just `1`) it is always just `model`
2. There's no dedicated `atomic-events` folder. It is replaced with unified `vendor=com.snowplowanalytics.snowplow/name=atomic/format=tsv/model=1`
3. There are no `shredded-types` or `shredded-tsv` either, all types are in the root of the folder.

Structure of the typical shredded folder now looks like following:

```text
run=2021-01-27-18-35-00/

   vendor=com.snowplowanalytics.snowplow/

     name=atomic/

       format=tsv/

         model=1/

   vendor=com.snowplowanalytics.snowplow/

     name=ad_click/

       format=json/

         model=1/

   vendor=nj.basjes/

     name=yauaa_context/

       format=tsv/

         model=1/

   shredding_complete.json

   _SUCCESS
```

## Caution

We consider this version a public beta. Although it has been carefully tested in sandbox environments showing significantly decreased AWS costs on associated infrastructure, it still haven't been used in production.

One known issue in this version is absence of protection against double-loading. If Loader receives the same SQS message multiple time (i.e. sent manually) - the same batch will be loaded multiple times.

We also reserve right to make other breaking API changes in next versions.

---

# Snowbridge batching model
> Understand how Snowbridge handles message batching across sources and targets for optimal throughput and performance.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/batching-model/

Snowbridge processes messages through a channel-based pipeline:

**Source → Transformer → Router → Targets**

Each stage operates concurrently, with backpressure propagated between stages via Go channels.

## Sources

Sources emit messages one at a time onto a shared channel. No batching occurs at the source level — sources are responsible only for reading and emitting messages.

## Transformer

The Transformer stage reads messages from the source channel and applies all configured transformations. It runs a worker pool of goroutines for concurrent processing. The pool scales automatically based on `GOMAXPROCS`, or can be set explicitly with the `worker_pool` option on the `transform {}` block.

```hcl
transform {

  worker_pool = 4



  use "spEnrichedToJson" {}

}
```

Filtered messages (those dropped by a filter transformation) are routed separately and never reach the targets.

## Router and batching

The Router receives transformed messages and accumulates them into per-target batches. A batch is sent to the target when any of the following conditions are met:

- The batch reaches `max_batch_messages`.
- The batch reaches `max_batch_bytes`.
- The `flush_period_millis` timer elapses (timer-based flush), ensuring batches do not wait indefinitely even during low-volume periods.

Messages that exceed `max_message_bytes` are detected by the Router before batching and are routed directly to the failure target as oversized messages.

The Router sends batches concurrently to each target, controlled by `max_concurrent_batches` per target.

## Configuring batching

From version 5.0.0, all targets support a `batching {}` configuration block. Each target has its own defaults; see the individual target configuration pages for the defaults.

```hcl
target {

  use "http" {

    url = "https://acme.com/x"



    batching {

      max_batch_messages     = 50

      max_batch_bytes        = 1048576

      max_message_bytes      = 1048576

      max_concurrent_batches = 5

      flush_period_millis    = 500

    }

  }

}
```

| Option                   | Description                                                                                    |
| ------------------------ | ---------------------------------------------------------------------------------------------- |
| `max_batch_messages`     | Maximum number of messages in a single batch.                                                  |
| `max_batch_bytes`        | Maximum total byte size of a batch.                                                            |
| `max_message_bytes`      | Maximum byte size of a single message. Messages exceeding this are sent to the failure target. |
| `max_concurrent_batches` | Number of batches written to the target concurrently (default: `5`).                           |
| `flush_period_millis`    | Milliseconds between timer-based batch flushes (default: `500`).                               |

---

# Snowbridge failure model
> Learn how Snowbridge handles target failures, oversized data, invalid data, transformation failures, and fatal errors.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/failure-model/

## Failure targets

When Snowbridge encounters an unrecoverable error — for example [oversized](#oversized-data) or [invalid](#invalid-data) data — it emits a [failed event](/docs/fundamentals/failed-events/) to the configured failure target. A failure target is configured the same way as a regular target; the only difference is that the destination receives failed events rather than the main data stream.

You can find more detail on setting up a failure target in the [configuration section](/docs/api-reference/snowbridge/configuration/targets/).

## Target failure

A target failure occurs when a write request to the destination fails or is rejected — for example, an HTTP 400 response.

Retry behavior is determined by the retry configuration. See the [retry configuration section](/docs/api-reference/snowbridge/configuration/retries/) for details.

The Kinesis target handles write throughput exceptions separately, with an in-built backoff and retry, rather than treating them as a standard failure type.

Each failed attempt is reported as a `target_failed` metric for monitoring purposes.

## Oversized data

Internally, application checks each message against the target's `max_message_bytes` limit before batching. Messages that exceed this limit produce a [size violation failed event](/docs/api-reference/failed-events/#size-violation) and are sent to the failure target.

Oversized message handling occurs before the message reaches the target, so no write is attempted to the main target.

## Invalid data

In the unlikely event that Snowbridge encounters data that the target cannot process (for example, empty data is invalid for PubSub), it creates a [generic error failed event](/docs/api-reference/failed-events/#generic-error), emits it to the failure target, and acks the original message.

The HTTP target may produce invalid failures in the following situations:

- A POST request body cannot be formed.
- The templating feature fails to template the data.
- A response rule configured with `type = "invalid"` matches the response.

Transformation failures are also treated as invalid (see [transformation failure](#transformation-failure)).

Invalid message writes to the failure target are reported as `failure_target_success` and `failure_target_failed` metrics.

## Transformation failure

Where a transformation throws an exception, Snowbridge treats the message as invalid. It creates a [generic error failed event](/docs/api-reference/failed-events/#generic-error), emits it to the failure target, and acks the original message.

Built-in transformations rarely fail when configured correctly. For scripting transformations, an exception is interpreted as meaning the transformation cannot process the data — construct and test your scripts accordingly.

## Fatal failure

A fatal failure is a scenario where continued processing is not possible and Snowbridge must stop.

There are three paths that lead to a fatal failure.

### Source read error

If Snowbridge cannot read from the source stream, it logs an error and exits. This is usually a misconfiguration of the source. When redeployed, the app resumes from the first unacked message, which may cause some duplicate sends to the target.

### Failure target write error

When invalid or oversized data cannot be written to the failure target, Snowbridge crashes. On redeploy, the app resumes from the last acked message. The likely impact is duplicated target sends, not data loss.

### Fatal response rule (graceful shutdown)

From version 5.0.0, the HTTP target supports a `fatal` response rule type. When a response matches a `fatal` rule, Snowbridge interprets this as a signal that the request can never succeed — for example, a 413 response indicating the payload is permanently too large. Rather than crashing immediately, Snowbridge completes all in-flight writes and flushes any remaining batches before shutting down gracefully.

```hcl
response_rules {

  rule {

    type       = "fatal"

    http_codes = [413]

  }

}
```

This provides a clean shutdown path distinct from a crash, ensuring in-flight data is not lost when the target signals an unrecoverable state.

---

# Snowbridge core concepts
> Understand Snowbridge architecture including sources, transformations, targets, batching, failure handling, and scaling strategies.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/

Snowbridge’s architecture is fairly simple: it receives data from one streaming technology (via [Sources](/docs/api-reference/snowbridge/concepts/sources/)), optionally runs filtering and transformation logic on them (message-by-message, via [Transformations](/docs/api-reference/snowbridge/concepts/transformations/)), and sends the data to another streaming technology or destination (via [Targets](/docs/api-reference/snowbridge/concepts/targets/)). If it is not possible to process or retry the data [as per the failure model](/docs/api-reference/snowbridge/concepts/failure-model/), it outputs a message to another destination (via [Failure Targets](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets)).

Where the source supports acking, Snowbridge only acks messages once the data is successfully sent to either the target or the failure target (in the case of unrecoverable failure). In the case of a recoverable failure — for example when the target is temporarily unavailable — Snowbridge will not ack the messages and will retry them once the source technology’s ack deadline has passed.

![Snowbridge architecture diagram showing an Input Stream flowing into a Stream Replicator containing a Source, a Filters and Transformations chain, and a Target. Valid messages flow from the Target to a Destination; invalid or oversized messages are routed to a Failure Target and on to a Failure Destination.](/assets/images/snowbridge-architecture-12bf56119494c1ca221081e869411e65.jpg)

## Operational details

Data is processed on an at-least-once basis, and there is no guarantee of order of messages. The application is designed to minimise duplicates as much as possible, but there isn’t a guarantee of avoiding them — for example if there’s a failure, it is possible for messages to be delivered without a successful response, and duplicates can occur.

---

# Scaling Snowbridge horizontally
> Scale Snowbridge horizontally across multiple instances with per-target concurrency controls and target provisioning for optimal throughput.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/scaling/

Snowbridge is built for a **horizontal scaling** model. You can deploy multiple instances consuming from the same source without additional configuration. No coordination setup is required between instances.

> **Note:** If you are using the Kinesis source, you must create a few DynamoDB tables as described in [the Kinesis source configuration section](/docs/api-reference/snowbridge/configuration/sources/kinesis/). Snowbridge uses these tables to coordinate multiple instances consuming from the same stream.

## Internal concurrency model

From version 5.0.0, Snowbridge uses a channel-based pipeline architecture:

**Source → Transformer → Router → Targets**

Each stage runs concurrently:

- The **Transformer** runs a worker pool of goroutines. The pool scales automatically based on `GOMAXPROCS`, or can be set explicitly via `worker_pool` in the `transform {}` block.
- The **Router** writes batches to targets concurrently, with concurrency controlled per target via `max_concurrent_batches`.

This architecture replaces the previous `concurrent_writes` source-level setting.

## Configuring concurrency

### Transformation concurrency

The `worker_pool` option controls how many goroutines process transformations in parallel. The default scales with `GOMAXPROCS`.

```hcl
transform {

  worker_pool = 8

  use "spEnrichedToJson" {}

}
```

### Target write concurrency

The `max_concurrent_batches` option on each target controls how many batch writes may be in flight simultaneously for that target.

```hcl
target {

  use "http" {

    url = "https://acme.com/x"



    batching {

      max_concurrent_batches = 10

    }

  }

}
```

Increasing `max_concurrent_batches` increases write throughput to the target, at the cost of higher resource usage and more open connections to the destination. Decrease it if the target is being overwhelmed.

## Scaling recommendations

Snowbridge should consume as much data as possible, as fast as possible. A backlog or traffic spike should cause CPU usage to increase noticeably. If spikes do not produce this behavior and there are no target retries or failures, try increasing `max_concurrent_batches`.

How you configure scaling depends on your infrastructure and use case. If you scale based on CPU usage, note that this metric is affected by the volume of data, the transformations configured, and — for scripting transformations — the content of the scripts.

> **Tip:** New Snowbridge releases sometimes improve efficiency significantly. After upgrading Snowbridge or changing the transformation configuration, monitor your scaling metrics to confirm that they behave as expected with the new version.

## Target scaling

Snowbridge sends data to the target as fast as resources allow. We recommend provisioning the target to scale with the expected throughput. If target writes fail, Snowbridge retries with exponential backoff. If a backlog builds up due to target downtime, overprovision the target until the backlog drains.

---

# Introduction to Snowbridge sources
> Configure Snowbridge sources to retrieve data from streams and forward them for processing with acking and concurrency controls.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/sources/

Sources deal with retrieving data from the input stream, and forwarding it for processing — once messages are either filtered or successfully sent, they are then acked (if the source technology supports acking). Otherwise, messages will be retrieved again by the source. Sources also have a setting which controls concurrency for the instance — `concurrent_writes`.

You can find more detail on setting up a source, in the [configuration section](/docs/api-reference/snowbridge/configuration/sources/).

---

# Introduction to Snowbridge targets
> Configure Snowbridge targets to validate, batch, and send data to destination streams with size and validity checks.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/targets/

Targets check for [validity and size restrictions](/docs/api-reference/snowbridge/concepts/failure-model/), [batch data](/docs/api-reference/snowbridge/concepts/batching-model/) where appropriate and send data to the destination stream.

You can find more detail on setting up a target, in the [configuration section](/docs/api-reference/snowbridge/configuration/targets/).

---

# Introduction to Snowbridge transformations and filters
> Transform and filter messages with built-in transformations for Snowplow data or custom JavaScript scripts for flexible data processing.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/transformations/

Transformations allow you to modify messages' data on the fly before they're sent to the destination. There are a set of built-in transformations, specifically for use with Snowplow data (for example transforming Snowplow enriched events to JSON), You can also configure a script to transform your data however you require - for example if you need to rename fields or change a field's format.

It's also possible to exclude messages (ie. not send them to the target) based on a condition, by configuring a special type of transformation called a filter. (Technically then, filters are transformations, but we sometimes refer to them as a separate concept for clarity). Again there are built-in filters to apply to Snowplow data, or you can provide a script to filter the data.

Transformations operate on a per-message basis, are chained together in the order configured, and the same type of transformation may be configured more than once. We recommend to place filters first for performance reasons. When transformations are chained together, the output of the first is the input of the second, however transformations may not depend on each other in any other way.

As an example of how transformations relate to each other - if you have a built-in filter with condition A, and a filter with condition B, you can arrange them one after another, so that the data must satisfy A AND B. But you can't arrange them to satisfy A OR B - because the outcome of each must be determined on their own.

The latter use case, and further nuanced use cases can, however, be achieved using scripting transformation (in the case of the latter example, a single script can perform both checks with an OR condition).

## Custom scripting transformations

Custom scripting transformations allow you to provide a script to transform the data, set the destination's partition key, or filter the data according to your own logic. For scripting, you can use Javascript. Snowbridge uses a runtime engine to run the script against the data. Scipts interface with the rest of the app via the `EngineProtocol` interface, which provides a means to pass data into the scripting layer, and return data from the scripting layer back to the app.

You can find more detail on setting up custom scripts [in the configuration section](/docs/api-reference/snowbridge/configuration/transformations/).

---

# Snowbridge configuration overview
> Configure Snowbridge using HCL format to define sources, transformations, targets, and monitoring for stream replication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/

Snowbridge is configured using [HCL](https://github.com/hashicorp/hcl). To configure Snowbridge, create your configuration in a file with `.hcl` extension, and set the `SNOWBRIDGE_CONFIG_FILE` environment variable to the path to your file. By default, the Snowbridge docker image uses `/tmp/config.hcl` as the config path - when using the docker images you can either mount your config file to `/tmp/config.hcl`, or mount it to a different path, and set the `SNOWBRIDGE_CONFIG_FILE` environment variable in your docker container to that path.

Inside the configuration, you can reference environment variables using the `env` object. For example, to refer to an environment variable named `MY_ENV_VAR` in your configuration, you can use `env.MY_ENV_VAR`. We recommend employing environment variables for any sensitive value, such as a password, as opposed to adding the value to the configuration verbatim.

For most options, Snowbridge uses blocks for configuration. The `use` keyword specifies what you'd like to configure - for example a kinesis source is configured using `source { use "kinesis" {...}}`.

For all configuration blocks except for transformations, you must provide only one block (or none, to use the defaults).

For transformations, you may provide 0 or more `transform` configuration blocks. They will be applied to the data, one after another, in the order they appear in the configuration. The exception to this is when a filter is applied and the filter condition is met - in this case the message will be acked and subsequent transformations will not be applied (neither will the data be sent to the destination).

Some application-level options are not contained in a block, instead they're top-level options in the configuration. For example, to set the log level of the application, we just set the top-level variable `log_level`.

If you do not provide a configuration, or provide an empty one, the application will use the defaults:

- `stdin` source;
- no transformations;
- `stdout` target;
- `stdout` failure target. There'll be no external statistics reporting or sentry error reporting.

## License

Since version 2.4.0, Snowbridge is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run Snowbridge, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {

  accept = true

}
```

## Example configuration

The below example is a complete configuration, which specifies a kinesis source, a builtin Snowplow filter (which may only be used if the input is Snowplow enriched data), a custom javascript transformation, and a pubsub target, as well as the statsD stats receiver, and sentry for error reporting.

In layman's terms, this configuration will read data from a kinesis stream, filter out any data whose `event_name` field is not `page_view`, run a custom Javascript script upon the data to change the `app_id` to `"1"`, and send the transformed page view data to pubsub. It will also send statistics about what it's doing to a statsD endpoint, and will send information about errors to a sentry endpoint.

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/overview-full-example.hcl&#xA;>)

---

# Snowbridge monitoring configuration
> Monitor Snowbridge with configurable logging, pprof profiling, StatsD metrics, and Sentry error reporting for observability.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/monitoring/

Snowbridge comes with configurable logging, [pprof](https://github.com/google/pprof) profiling, [statsD](https://www.datadoghq.com/statsd-monitoring) statistics and [Sentry](https://sentry.io/welcome/) integrations to ensure that you know what's going on.

## Logging

Use the `log_level` parameter to specify the log level.

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/monitoring/log-level-example.hcl&#xA;>)

## Sentry Configuration

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/monitoring/sentry-example.hcl&#xA;>)

## StatsD stats receiver configuration

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/monitoring/statsd-example.hcl&#xA;>)

## End-to-end latency configuration

Snowplow Enriched data only:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/metrics/e2e-latency-example.hcl&#xA;>)

## Metric definitions

Snowbridge sends the following metrics to StatsD:

| Metric                   | Description                                                                                                                                 |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `target_success`         | Events successfully sent to the target.                                                                                                     |
| `target_failed`          | Events that failed to reach the target and will be retried.                                                                                 |
| `target_request_count`   | Number of requests successfully sent to the target.                                                                                         |
| `message_filtered`       | Events filtered out via transformation.                                                                                                     |
| `failure_target_success` | Invalid events (not retryable) successfully sent to the failure target.                                                                     |
| `failure_target_failed`  | Invalid events that failed to reach the failure target. Snowbridge crashes in this scenario.                                                |
| `min_processing_latency` | Minimum time between entering Snowbridge and writing to the target.                                                                         |
| `max_processing_latency` | Maximum time between entering Snowbridge and writing to the target.                                                                         |
| `min_message_latency`    | Minimum time between entering the source stream and writing to the target.                                                                  |
| `max_message_latency`    | Maximum time between entering the source stream and writing to the target.                                                                  |
| `min_transform_latency`  | Minimum time for a transformation to complete.                                                                                              |
| `max_transform_latency`  | Maximum time for a transformation to complete.                                                                                              |
| `min_filter_latency`     | Minimum time between entering Snowbridge and being filtered out.                                                                            |
| `max_filter_latency`     | Maximum time between entering Snowbridge and being filtered out.                                                                            |
| `min_request_latency`    | Minimum time for a target write request to complete.                                                                                        |
| `max_request_latency`    | Maximum time for a target write request to complete.                                                                                        |
| `min_e2e_latency`        | Minimum time between Snowplow collector timestamp and completing the target write. Enabled via configuration — Snowplow enriched data only. |
| `max_e2e_latency`        | Maximum time between Snowplow collector timestamp and completing the target write. Enabled via configuration — Snowplow enriched data only. |

---

# Snowbridge retry behavior configuration (beta)
> Configure retry behavior for transient and setup failures with exponential backoff and configurable max attempts.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/retries/

> **Note:** This feature was added in version 3.0.0
>
> This feature is in beta status because we may make breaking changes in future versions.

This feature allows you to configure the retry behavior when the target encounters a failure in sending the data. There are three types of failure you can define:

A **transient failure** is a failure which we expect to succeed again on retry. For example, some temporary network error. Typically, you would configure a short backoff for this type of failure. When we encounter a transient failure, we keep processing the rest of the data as normal, under the expectation that everything is operating as normal. The failed data is retried after a backoff.

A **setup failure** is one we don't expect to be immediately resolved, for example an incorrect address, or an invalid API key. Typically, you would configure a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (e.g. permissions need to be granted). Setup errors will be retried up to the configured `max_attempts` before the app crashes.

A **throttle failure** (added in version 4.0.0) is a special type of failure that indicates the target is rate limiting requests. This is handled separately from transient errors to allow different retry behavior - typically with longer delays to respect rate limits.

As of version 3.0.0, only the http target can be configured to return setup and throttle errors, via response rules - see [the http target configuration section](/docs/api-reference/snowbridge/configuration/targets/http/). For all other targets, all errors returned will be considered transient, and behavior can be configured using the `transient` block of the retry configuration.

Retries will be attempted with an exponential backoff. In other words, on each subsequent failure, the backoff time will double. You can configure transient failures to be retried indefinitely by setting `max_attempts` to 0.

As of version 4.0.0, you can configure transient failures to be sent to the failure target after reaching `max_attempts` by setting `invalid_after_max` to `true`.

## Configuration options

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/retry-example.hcl&#xA;>)

---

# Configure HTTP as a Snowbridge source
> Configure HTTP source for Snowplow Snowbridge to receive data over HTTP endpoints for experimental stream ingestion.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/http/

> **Note:** This source was added in version 3.6.2
>
> This source is experimental and not recommended for production use.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/http-minimal-example.hcl&#xA;>)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/http-full-example.hcl&#xA;>)

---

# Snowbridge source configuration
> Configure Snowbridge sources including stdin, Kafka, Kinesis, PubSub, SQS, and HTTP for stream ingestion.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/

**Stdin source** is the default. We also support Kafka, Kinesis, PubSub, SQS and experimental HTTP sources.

Stdin source simply treats stdin as the input. It has one optional configuration to set the concurrency.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/stdin-minimal-example.hcl&#xA;>)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/stdin-full-example.hcl&#xA;>)

---

# Configure Kafka as a Snowbridge source
> Configure Kafka source for Snowplow Snowbridge to read data from Kafka topics with authentication and consumer group settings.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/kafka/

Authentication is done by providing valid credentials in the configuration.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/kafka-minimal-example.hcl&#xA;>)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/kafka-full-example.hcl&#xA;>)

---

# Configure Kinesis as a Snowbridge source
> Configure Kinesis source for Snowplow Snowbridge to read from AWS Kinesis streams with DynamoDB checkpointing and authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/kinesis/

> **Note:** To use this source, you need the AWS-specific version of Snowbridge that can only be run on AWS. See [the page on Snowbridge distributions](/docs/api-reference/snowbridge/getting-started/) for more information.

## Authentication

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the stream.

## Setup

The AWS kinesis source requires the additional setup of a set of dynamoDB tables for checkpointing purposes. To set up a kinesis source, you will need to:

1. Configure the above required variables in the HCL file.
2. Create three DynamoDB tables which will be used for checkpointing the progress of the replicator on the stream (_Note_: details below)

Under the hood we are using a fork of the [Kinsumer](https://github.com/snowplow-devops/kinsumer) library which has defined this DynamoDB table structure - these tables need to be created by hand before the application can launch.

| TableName                                | DistKey        |
| ---------------------------------------- | -------------- |
| `${SOURCE_KINESIS_APP_NAME}_clients`     | ID (String)    |
| `${SOURCE_KINESIS_APP_NAME}_checkpoints` | Shard (String) |
| `${SOURCE_KINESIS_APP_NAME}_metadata`    | Key (String)   |

Assuming your AWS credentials have sufficient permission for Kinesis and DynamoDB, your consumer should now be able to run when you launch the executable.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/kinesis-minimal-example.hcl&#xA;>)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/kinesis-full-example.hcl&#xA;>)

---

# Configure Pub/Sub as a Snowbridge source
> Configure PubSub source for Snowplow Snowbridge to read from GCP Pub/Sub subscriptions with service account authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/pubsub/

Authentication is done using a [GCP Service Account](https://cloud.google.com/docs/authentication/application-default-credentials#attached-sa). Create a service account credentials file, and provide the path to it via the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.

Snowbridge connects to PubSub using [Google's Go Pubsub sdk](https://cloud.google.com/go/pubsub), which establishes a grpc connection with TLS encryption.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/pubsub-minimal-example.hcl&#xA;>)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/pubsub-full-example.hcl&#xA;>)

---

# Configure SQS as a Snowbridge source
> Configure SQS source for Snowplow Snowbridge to read from AWS SQS queues with IAM authentication and message handling.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/sqs/

Read data from an SQS queue.

## Authentication

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the stream.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/sqs-minimal-example.hcl&#xA;>)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/sources/sqs-full-example.hcl&#xA;>)

---

# Configure EventHub as a Snowbridge target
> Configure EventHub target for Snowplow Snowbridge to write data to Azure Event Hubs with namespace and environment authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/eventhub/

Authentication for the EventHub target is done by configuring any valid combination of the environment variables [listed in the Azure Event Hubs Client documentation](https://pkg.go.dev/github.com/Azure/azure-event-hubs-go#NewHubWithNamespaceNameAndEnvironment).

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/eventhub-minimal-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target. Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/eventhub-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure HTTP as a Snowbridge target
> Configure HTTP target for Snowplow Snowbridge to send data over HTTP with authentication, OAuth2, request templating, and response rules.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/http/

> **Note:** Version 4.0.0 makes breaking changes to the HTTP target. Details on migrating can be found [in the migration guide](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-4-X-X/)

> **Note:** Version 5.0.0 makes breaking changes to the HTTP target. Details on migrating can be found [in the migration guide](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-5-X-X/)

## Basic authentication

Where basicauth is used, it may be configured using the `basic_auth_username` and `basic_auth_password` options. Where an authorization header is used, it may be set via the `headers` option.

We recommend using environment variables for sensitive values - which can be done via HCL's native `env.MY_ENV_VAR` format (as seen below).

TLS may be configured by providing the `key_file`, `cert_file` and `ca_file` options with paths to the relevant TLS files.

## OAuth2

Snowbridge supports sending authorized requests to OAuth2-compliant HTTP targets. This can be enabled by setting `oauth2_client_id`, `oauth2_client_secret`, `oauth2_refresh_token` (these three are long-lived credentials used to generate short-lived bearer access tokens), and `oauth2_token_url` (which is the URL of the authorization server providing access tokens).

Like in the case of basic authentication, we recommend using environment variables for sensitive values.

## Dynamic headers

> **Note:** This feature was added in version 2.3.0

When enabled, this feature attaches a header to the data according to what your transformation provides in the `HTTPHeaders` field of `engineProtocol`. Data is batched independently per each dynamic header value before requests are sent.

## Request templating

> **Note:** This feature was added in version 3.0.0

This feature allows you to provide a [Golang text template](https://pkg.go.dev/text/template) to construct a request body from a batch of data. This feature should be useful in constructing requests to send to an API, for example.

Input data must be valid JSON, any message that fails to be marshaled to JSON will be treated as invalid and sent to the failure target. Equally, if an attempt to template a batch of data results in an error, then all messages in the batch will be considered invalid and sent to the failure target.

Where the dynamic headers feature is enabled, data is split into batches according to the provided header value, and the templater will operate on each batch separately.

### Helper functions

In addition to all base functions available in the Go text/template package, the following custom functions are available for convenience:

`prettyPrint` - Because the input to the templater is a Go data structure, simply providing a reference to an object field won't output it as JSON. `prettyPrint` converts the data to prettified JSON. Use it wherever you expect a JSON object in the output. This is compatible with any data type, but it shouldn't be necessary if the data is not an object.

`env` - Allows you to set and refer to an env var in your template. Use it when your request body must contain sensitive data, for example an API key.

### Template example

The following example provides an API key via environment variable, and iterates the batch to provide JSON-formatted data one by one into a new key, inserting a comma before all but the first event.

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/http-template-full-example.file&#xA;>)

## Response rules (beta)

> **Note:** This feature was added in version 3.0.0
>
> This feature is in beta status because we may make breaking changes in future versions.
>
> **Breaking change in version 4.0.0**: Response rules are now evaluated in the order they are defined in the configuration. Rules must specify a `type` attribute to distinguish between invalid, setup, and throttle errors, rather than organizing them in separate `invalid` and `setup` blocks.

Response rules allow you to configure how the app deals with failures in sending the data. You can configure a response code and an optional string match on the response body to determine how a failure response is handled. Response codes between 200 and 299 are considered successful, and are not handled by this feature.

**Response rules are evaluated in the order they are defined in your configuration.** The first matching rule determines how the error is categorized.

There are four categories of failure:

`invalid` means that the data is considered incompatible with the target for some reason. For example, you may have defined a mapping for a given API, but the event happens to have null data for a required field. In this instance, retrying the data won't fix the issue, so you would configure an invalid response rule, identifying which responses indicate this scenario.

Data that matches an invalid response rule is sent to the failure target.

`setup` means that this error is not retryable, but is something which can only be resolved by a change in configuration or a change to the target. An example of this is an authentication failure - retrying won't fix the issue; the resolution is to grant the appropriate permissions, or provide the correct API key.

Data that matches a setup response rule is handled by a retry as determined in the `setup` configuration block of [retry configuration](/docs/api-reference/snowbridge/configuration/retries/).

`throttle` (added in version 4.0.0) is a special type of error that indicates the target is rate limiting requests. This is handled separately from transient errors to allow different retry behavior - typically with longer delays to respect rate limits.

Data that matches a throttle response rule is handled by a retry as determined in the `throttle` configuration block of [retry configuration](/docs/api-reference/snowbridge/configuration/retries/).

`transient` errors are everything else - we assume that the issue is temporary and retrying will resolve the problem. There is no explicit configuration for transient - rather, anything that is not configured as one of the other types is considered transient.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/http-minimal-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target. Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/http-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

## Example for Google Tag Manager Server Side

You can use the HTTP target to send events to Google Tag Manager Server Side, where the [Snowplow Client tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/) is installed.

To do this, you will need to include a [transformation](/docs/api-reference/snowbridge/concepts/transformations/) that converts your events to JSON — [`spEnrichedToJson`](/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedToJson/).

Here’s an example configuration. Replace `<your-gtm-host>` with the hostname of your Google Tag Manager instance, and — optionally — `<preview-token>` with your preview mode token.

```hcl
target {

  use "http" {

    url                        = "https://<your-gtm-host>/com.snowplowanalytics.snowplow/enriched"

    request_timeout_in_seconds = 5

    content_type               = "application/json"



    # this line is optional, in case you want to send events to GTM Preview Mode

    headers                    = "{\"x-gtm-server-preview\": \"<preview-token>\"}"

  }

}



transform {

  use "spEnrichedToJson" {}

}
```

---

# Snowbridge target configuration
> Configure Snowbridge targets including stdout, EventHub, HTTP, Kafka, Kinesis, PubSub, and SQS for stream output.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/

**Stdout target** is the default. We also support EventHub, HTTP, Kafka, Kinese, PubSub, and SQS targets.

Stdout target doesn't have any configurable options - when configured it simply outputs the messages to stdout.

## Configuration options

Here is an example of the configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/stdout-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use `failure_target` instead of `target`.

## Batching

From version 5.0.0, all targets support configurable batching. Batching options are specified in a `batching {}` block inside the target's `use` block.

```hcl
target {

  use "http" {

    url = "https://acme.com/x"



    batching {

      max_batch_messages     = 50

      max_batch_bytes        = 1048576

      max_message_bytes      = 1048576

      max_concurrent_batches = 5

      flush_period_millis    = 500

    }

  }

}
```

Each target has its own defaults for batching options. See the individual target pages for the defaults.

| Batching option          | Description                                                                                    |
| ------------------------ | ---------------------------------------------------------------------------------------------- |
| `max_batch_messages`     | Maximum number of messages in a single batch sent to the target.                               |
| `max_batch_bytes`        | Maximum total byte size of a batch.                                                            |
| `max_message_bytes`      | Maximum byte size of a single message. Messages exceeding this are sent to the failure target. |
| `max_concurrent_batches` | Number of batches written concurrently (default: `5`).                                         |
| `flush_period_millis`    | Interval in milliseconds between timer-based batch flushes (default: `500`).                   |

If either of `max_batch_messages` or `max_batch_bytes` is reached before `flush_period_millis` elapses, the batch is sent immediately.

---

# Configure Kafka as a Snowbridge target
> Configure Kafka target for Snowplow Snowbridge to write data to Kafka topics with SASL authentication and TLS encryption.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/kafka/

Where SASL is used, it may be enabled via the `enable_sasl`, `sasl_username`, and `sasl_password` and `sasl_algorithm` options.

we recommend using environment variables for sensitive values - which can be done via HCL's native `env.MY_ENV_VAR` format (as seen below).

TLS may be configured by providing the `key_file`, `cert_file` and `ca_file` options with paths to the relevant TLS files.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/kafka-minimal-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/kafka-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure Kinesis as a Snowbridge target
> Configure Kinesis target for Snowplow Snowbridge to write data to AWS Kinesis streams with throttle retry handling and IAM authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/kinesis/

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the stream.

## Throttle retries

As of 2.4.2, the kinesis target handles kinesis write throughput exceptions separately from all other errors and failures. It will back off and retry only the throttled records on an initial back off of 50ms, increasing by 50ms each time, until there are no more throttle errors.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/kinesis-minimal-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target. Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/kinesis-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure Pub/Sub as a Snowbridge target
> Configure PubSub target for Snowplow Snowbridge to write data to GCP Pub/Sub topics with service account authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/pubsub/

Authentication is done using a [GCP Service Account](https://cloud.google.com/docs/authentication/application-default-credentials#attached-sa). Create a service account credentials file, and provide the path to it via the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.

Snowbridge connects to PubSub using [Google's Go Pubsub sdk](https://cloud.google.com/go/pubsub), which establishes a grpc connection with TLS encryption.

## Configuration options

The PubSub Target has only two required options, and no optional ones.

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/pubsub-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure SQS as a Snowbridge target
> Configure SQS target for Snowplow Snowbridge to write data to AWS SQS queues with IAM authentication and message attributes.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/sqs/

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the queue.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/sqs-minimal-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/targets/sqs-full-example.hcl&#xA;>)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Snowbridge telemetry configuration
> Enable or disable telemetry for Snowbridge with user-provided identifiers and privacy controls.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/telemetry/

You can read about our telemetry principles [here](/docs/get-started/self-hosted/telemetry/).

To enable telemetry:

```hcl
# Optional. Set to true to disable telemetry.

disable_telemetry = false



# Optional. An identifier to associate with telemetry data.

user_provided_id = "elmer.fudd@acme.com"
```

To disable telemetry:

```hcl
# Optional. Set to true to disable telemetry.

disable_telemetry = false
```

---

# Snowbridge base64Decode transformation
> Base64 decode message data from base64 byte array to decoded byte array representation for Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/base64Decode/

Introduced in version 2.1.0

`base64Decode`: Base64 decodes the message's data.

This transformation base64 decodes the message's data from a base64 byte array, to a byte array representation of the decoded data.

`base64Decode` has no options.

## Configuration options

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/builtin/base64Decode-minimal-example.hcl&#xA;>)

---

# Snowbridge base64Encode transformation
> Base64 encode message data to base64 byte array representation for Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/base64Encode/

Introduced in version 2.1.0

`base64Encode`: Base64 encodes the message's data.

This transformation base64 encodes the message's data to a base 64 byte array.

`base64Encode` has no options.

## Configuration options

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/builtin/base64Encode-minimal-example.hcl&#xA;>)

---

# Built-in Snowbridge transformations
> Use built-in Snowbridge transformations for Snowplow data including enriched filtering, JSON conversion, and base64 encoding.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/

Snowbridge includes several configurable built-in transformations.

| Transformation                  | Functionality                                                                          | Snowplow data only |
| ------------------------------- | -------------------------------------------------------------------------------------- | ------------------ |
| `base64Decode`                  | Base64-decodes the message's data.                                                     |                    |
| `base64Encode`                  | Base64 encodes the message's data.                                                     |                    |
| `jq`                            | Runs a `jq` command on the message data, and outputs the result of the command.        |                    |
| `jqFilter`                      | Filters messages based on the output of a `jq` command.                                |                    |
| `spEnrichedFilter`              | Filters messages based on a regex match against an atomic field.                       | ✅                  |
| `spEnrichedFilterContext`       | Filters messages based on a regex match against a field in an entity.                  | ✅                  |
| `spEnrichedFilterUnstructEvent` | Filters messages based on a regex match against a field in a custom event.             | ✅                  |
| `spEnrichedSetPk`               | Sets the message's destination partition key.                                          | ✅                  |
| `spEnrichedToJson`              | Transforms a message's data from Snowplow enriched tsv string format to a JSON object. | ✅                  |
| `spGtmssPreview`                | Attaches a GTM SS preview mode header                                                  | ✅                  |

---

# Snowbridge jq transformation
> Run jq commands on message data to transform JSON structures with custom queries and helper functions, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/jq/

> **Note:** This transformation was added in version 3.0.0.

[jq](https://github.com/jqlang/jq) is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the [gojq](https://github.com/itchyny/gojq) package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the [tutorial](https://jqlang.github.io/jq/tutorial/), and [full reference manual](https://jqlang.github.io/jq/manual/). [This open-source jq playground tool](https://jqplay.org/) may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are [some small differences](https://github.com/itchyny/gojq?tab=readme-ov-file#difference-to-jq) between jq and gojq.

`jq` runs a jq command on the message data, and outputs the result of the command. While jq supports multi-element results, commands must output only a single element - this single element can be an array data type.

If the provided jq command results in an error, the message will be considered invalid, and will be sent to the failure target.

The minimal example here returns the input data as a single element array, and the full example maps the data to a new data structure.

The jq transformation will remove any keys with null values from the data.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/builtin/jq-minimal-example.hcl&#xA;>)

Every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/builtin/jq-full-example.hcl&#xA;>)

## Helper functions

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

- `epoch` converts a Go `time.Time` timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as `time.Time`. This function can be chained with jq native functions to get past this limitation. For example:

```text
{ foo: .collector_tstamp | epoch | todateiso8601 }
```

- `epochMillis` converts a Go `time.Time` timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the `epoch` function truncates to seconds, and the `epochMillis` function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.

```text
{ foo: .collector_tstamp | epochMillis }
```

- `hash(algorithm, salt)` hashes the input value. To use unsalted hash, pass an empty string for salt. Salt may be provided as an environment variable using hcl syntax.

The following hash algorithms are supported:

- `sha1` - SHA-1 hash (160 bits)
- `sha256` - SHA-256 hash (256 bits)
- `md5` - MD5 hash (128 bits)

```text
{ foo: .user_id |  hash("sha1"; "${env.SHA1_SALT}") }
```

---

# Snowbridge jqFilter transformation
> Filter messages using jq commands that return boolean results to keep or discard messages, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/

> **Note:** This transformation was added in version 3.0.0.

[jq](https://github.com/jqlang/jq) is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the [gojq](https://github.com/itchyny/gojq) package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the [tutorial](https://jqlang.github.io/jq/tutorial/), and [full reference manual](https://jqlang.github.io/jq/manual/). [This open-source jq playground tool](https://jqplay.org/) may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are [some small differences](https://github.com/itchyny/gojq?tab=readme-ov-file#difference-to-jq) between jq and gojq.

`jqFilter` filters messages based on the output of a jq command which is run against the data. The provided command must return a boolean result. `false` filters the message out, `true` keeps it.

If the provided jq command returns a non-boolean value error, or results in an error, then the message will be considered invalid, and will be sent to the failure target.

## Configuration options

This example filters out all data that doesn't have an `app_id` key.

Minimal configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/builtin/jqFilter-minimal-example.hcl&#xA;>)

Every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/builtin/jqFilter-full-example.hcl&#xA;>)

## Filtering examples

The following examples demonstrate common filtering patterns for Snowplow enriched events. You can filter Snowplow enriched events based on any field in the data.

Match an atomic field to any value from a list:

```hcl
transform {

  use "jqFilter" {



    # Keep only web and mobile data

    jq_command = <<JQEOT

    .platform | IN("web", "mobile")

JQEOT



    snowplow_mode = true

  }

}
```

Regex match against a singular entity:

```hcl
transform {

  use "jqFilter" {



    # Keep only "environment" matching a regex in custom event

    # `// ""` is needed as `null` is not regex compatible

    jq_command = <<JQEOT

    .contexts_com_acme_env_context_1.environment // "" | test("^prod")

JQEOT



    snowplow_mode = true

  }

}
```

Filter where any entity entry matches the condition:

```hcl
transform {

  use "jqFilter" {



    # Keep if any entry's environment matches one of two values:

    jq_command = <<JQEOT

    .contexts_com_acme_env_context_1 | any(.[]; .environment == "prod" or .environment == "staging")

JQEOT



    snowplow_mode = true

  }

}
```

Filter for an exact match on a self-describing event field:

```hcl
transform {

  use "jqFilter" {

    # Keep only "sku" of "test-data" in custom event

    jq_command = <<JQEOT

    .unstruct_event_com_acme_my_custom_event_1.sku == "test-data"

JQEOT



    snowplow_mode = true

  }

}
```

## Helper Functions

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

- `epoch` converts a Go `time.Time` timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as `time.Time`. This function can be chained with jq native functions to get past this limitation. For example:

```text
{ foo: .collector_tstamp | epoch | todateiso8601 }
```

- `epochMillis` converts a Go `time.Time` timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the `epoch` function truncates to seconds, and the `epochMillis` function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.

```text
{ foo: .collector_tstamp | epochMillis }
```

- `hash(algorithm, salt)` hashes the input value. To use unsalted hash, pass an empty string for salt. Salt may be provided as an environment variable using hcl syntax.

The following hash algorithms are supported:

- `sha1` - SHA-1 hash (160 bits)
- `sha256` - SHA-256 hash (256 bits)
- `md5` - MD5 hash (128 bits)

```text
{ foo: .user_id |  hash("sha1"; "${env.SHA1_SALT}") }
```

---

# Snowbridge spEnrichedFilter transformation
> Filter Snowplow enriched events based on regex matches against atomic fields with keep or drop actions, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedFilter/

> **Note:** This transformation is deprecated and can result in unexpected behavior when matching integers. Use the [`jqFilter`](/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/) transformation instead, which provides more robust and flexible filtering capabilities.

`spEnrichedFilter`: Specific to Snowplow data. Filters messages based on a regex match against an atomic field.

This transformation is for use on base-level atomic fields, rather than fields from contexts, or custom events — which can be achieved with `spEnrichedFilterContext` and `spEnrichedFilterUnstructEvent`.

Filters can be used in one of two ways, which is determined by the `filter_action` option. `filter_action` determines the behavior of the app when the regex provided evaluates to `true`. If it's set to `"keep"`, the app will complete the remaining transformations and send the message to the destination (unless a subsequent filter determines otherwise). If it's set to `"drop"`, the message will be acked and discarded, without continuing to the next transformation or target.

This example filters out all data whose `platform` value does not match either `web` or `mobile`.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilter-minimal-example.hcl&#xA;>)

Every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilter-full-example.hcl&#xA;>)

---

# Snowbridge spEnrichedFilterContext transformation
> Filter Snowplow enriched events based on regex matches against entity fields using jsonpath notation, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedFilterContext/

> **Note:** This transformation is deprecated and can result in unexpected behavior when matching integers. Use the [`jqFilter`](/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/) transformation instead, which provides more robust and flexible filtering capabilities.

`spEnrichedFilterContext`: Specific to Snowplow data. Filters messages based on a regex match against a field in an entity.

This transformation is for use on fields from entities (contexts).

Note that if the same context is present in the data more than once, one instance of a match is enough for the regex condition to be considered a match — and the message to be kept.

The full parsed context name must be provided, in camel case, in the format returned by the Snowplow analytics SDK: `contexts_{vendor}_{name}_{major version}` — for example `contexts_nl_basjes_yauaa_context_1`.

The path to the field to be matched must then be provided as a jsonpath (dot notation and square braces only) — for example `test1.test2[0].test3`.

Filters can be used in one of two ways, which is determined by the `filter_action` option. `filter_action` determines the behavior of the app when the regex provided evaluates to `true`. If it's set to `"keep"`, the app will complete the remaining transformations and send the message to the destination (unless a subsequent filter determines otherwise). If it's set to `"drop"`, the message will be acked and discarded, without continuing to the next transformation or target.

The below example keeps messages which contain `prod` in the `environment` field of the `contexts_com_acme_env_context_1` context. Note that the `contexts_com_acme_env_context_1` context is attached more than once, if _any_ of the values at `dev` don't match `environment`, the message will be kept.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterContext-minimal-example.hcl&#xA;>)

Every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterContext-full-example.hcl&#xA;>)

---

# Snowbridge spEnrichedFilterUnstructEvent transformation
> Filter Snowplow enriched events based on regex matches against self-describing event fields using jsonpath notation, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedFilterUnstructEvent/

> **Note:** This transformation is deprecated and can result in unexpected behavior when matching integers. Use the [`jqFilter`](/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/) transformation instead, which provides more robust and flexible filtering capabilities.

`spEnrichedFilterUnstructEvent`: Specific to Snowplow data. Filters messages based on a regex match against a field in a custom event.

This transformation is for use on fields from custom events.

The event name must be provided as it appears in the `event_name` field of the event (eg. `add_to_cart`).

Optionally, a regex can be provided to match against the stringified version of the event (eg. `1-*-*`)

The path to the field to match against must be provided as a jsonpath (dot notation and square braces only) — for example `test1.test2[0].test3`.

Filters can be used in one of two ways, which is determined by the `filter_action` option. `filter_action` determines the behavior of the app when the regex provided evaluates to `true`. If it's set to `"keep"`, the app will complete the remaining transformations and send the message to the destination (unless a subsequent filter determines otherwise). If it's set to `"drop"`, the message will be acked and discarded, without continuing to the next transformation or target.

## Configuration options

This example keeps all events whose `add_to_cart` event data at the `sku` field matches `test-data`.

Minimal configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterUnstructEvent-minimal-example.hcl&#xA;>)

Every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterUnstructEvent-full-example.hcl&#xA;>)

---

# Snowbridge spEnrichedSetPk transformation
> Set destination partition key for Snowplow enriched events using atomic field values, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedSetPk/

`spEnrichedSetPk`: Specific to Snowplow data. Sets the message's destination partition key to an atomic field from a Snowplow Enriched tsv string. The input data must be a valid Snowplow enriched TSV.

## Configuration options

`SpEnrichedSetPk` only takes one option — the field to use for the partition key.

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedSetPk-minimal-example.hcl&#xA;>)

Note: currently, setting partition key to fields in custom events and contexts is unsupported.

---

# Snowbridge spEnrichedToJson transformation
> Transform Snowplow enriched TSV data to JSON format using the Go Analytics SDK, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedToJson/

`spEnrichedToJson`: Specific to Snowplow data. Transforms a message's data from Snowplow Enriched tsv string format to a JSON object. The input data must be a valid Snowplow enriched TSV.

`spEnrichedToJson` has no options.

## Configuration options

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedToJson-minimal-example.hcl&#xA;>)

The transformation to JSON is done via the [analytics SDK](/docs/api-reference/analytics-sdk/) logic, specifically in this case the [Golang analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-go/).

In brief, the relevant logic here is that:

- If a field is not populated in the original event, it won't have a key in the resulting JSON

- In the TSV, there's a separate field for `contexts` (sent via tracker), and `derived_contexts` (attached during enrichment). In the analytics SDK, there is one key per context, regardless of which type. (Technically, it's one key per major version of a context. So if you had a 1.0.0 and a 2.0.0 of the same one, you'd have two keys).

---

# Snowbridge spGtmssPreview transformation
> Extract GTM Server Side preview mode header from Snowplow context for debugging with GTM preview mode, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spGtmssPreview/

> **Note:** This transformation was added in version 2.3.0

`spGtmssPreview`: Specific to Snowplow data. Extracts a value from the `x-gtm-server-preview` field of a [preview mode context](https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.tag-manager.server-side/preview_mode/jsonschema/1-0-0), and attaches it as the GTM SS preview mode header, to enable easier debugging using GTM SS preview mode.

Only one preview mode context should be sent at a time.

> **Note:** As of version 3.0.0:
>
> Invalid preview headers sent to GTM SS can result in requests failing, which may be problematic. There is insufficient information available about the values to allow us to confidently validate them, but we do two things to avoid this problem.
>
> First, we validate to ensure that the value is a valid base64 string. Second, we compare the age of the event (based on `collector_tstamp`) to ensure it is under a configurable timeout age. If either of these conditions fail, we treat the message as invalid, and output to the failure target.

## Configuration options

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spGtmssPreview-minimal-example.hcl&#xA;>)

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/snowplow-builtin/spGtmssPreview-full-example.hcl&#xA;>)

---

# Snowbridge Script transformation examples
> View example Snowbridge JavaScript transformations for Snowplow and non-Snowplow data including filtering, field modification, and partition keys.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/

Examples showing the transformation of Snowplow or non-Snowplow data.

## Non-Snowplow data

For this example, the input data is a json string which looks like this:

```json
{

  "name": "Bruce",

  "id": "b47m4n",

  "batmobileCount": 1

}
```

The script filters out any data with a `batmobileCount` less than 1, otherwise it updates the Data's `name` field to "Bruce Wayne", and sets the PartitionKey to the value of `id`:

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/examples/js-non-snowplow-script-example.js&#xA;>)

The configuration for this script is:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/examples/js-non-snowplow-config-example.hcl&#xA;>)

## Snowplow data

For this example, the input data is a valid Snowplow TSV event - so we can enable `snowplow_mode`, which will convert the data to a JSON before passing it to the script as a JSON object.

The script below filters out non-web data, based on the `platform` value, otherwise it checks for a `user_id` value, setting a new `uid` field to that value if it's found, or `domain_userid` if not.

It also sets the partitionKey to `app_id`.

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/examples/js-snowplow-script-example.js&#xA;>)

The configuration for this script is:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/examples/js-snowplow-config-example.hcl&#xA;>)

---

# Custom Snowbridge script transformations
> Create custom JavaScript Snowbridge transformations to modify data, filter messages, set partition keys, and add HTTP headers.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/

Custom transformation scripts may be defined in Javascript and provided to Snowbridge.

## The scripting interface

The script must define a main function with a single argument. Snowbridge will pass the `engineProtocol` data structure as the argument:

```go
type engineProtocol struct {

    FilterOut    bool

    PartitionKey string

    Data         interface{}

    HTTPHeaders  map[string]string

}
```

This structure is represented as an object in the script engine, and serves as both the input and output of the script.

Scripts must define a `main` function with a single input argument (JSDoc for type information is optional):

```js
/**

 * @typedef {object} EngineProtocol

 * @property {boolean} FilterOut

 * @property {string} PartitionKey

 * @property {(string | Object.<string, ?>)} Data

 * @property {Object.<string, string>} HTTPHeaders

 */



/**

 * @param {EngineProtocol} input

 * @return {Partial<EngineProtocol>}

 */

function main(input) {

    return input

}
```

## Accessing data

Scripts can access the message Data at `input.Data`, and can return modified data by returning it in the `Data` field of the output. Likewise for the partition key to be used for the destination - `input.PartitionKey` and the `PartitionKey` field of the output.

By default, the input's `Data` field will be a string in [enriched TSV format](/docs/pipeline/enriched-tsv-format/). This can be changed with the [SpEnrichedToJson](/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedToJson/) transform, or the Javascript transformation itself has a `snowplow_mode` option, which transforms the data to an object first.

The output of the script must be an object which maps to engineProtocol.

### `snowplow_mode`

`snowplow_mode` uses the [Go Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-go/) to parse the TSV fields into an object suitable for working with in Go. The result of the [`ParsedEvent.ToMap()`](https://pkg.go.dev/github.com/snowplow/snowplow-golang-analytics-sdk/analytics#ParsedEvent.ToMap) method is the input to your transform function.

The [keys of the resulting map](https://github.com/snowplow/snowplow-golang-analytics-sdk/blob/a3430fbe576483d615b713120cfb5e443897d572/analytics/mappings.go#L153) are defined by the Analytics SDK. Values are primitives (string, number, boolean) or, for timestamps (e.g. `derived_tstamp`, `collector_tstamp`), [`time.Time`](https://pkg.go.dev/time#Time) objects.

> **Tip:** To work with native Javascript [Date](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date) objects for timestamps, construct one via `new Date(ts.UnixMilli())` where `ts` is a `time.Time` instance.
>
> When returned as part of `Data`, `time.Time` instances will [serialize in JSON](https://pkg.go.dev/time#Time.MarshalJSON) as [RFC3339](https://www.rfc-editor.org/rfc/rfc3339.html) strings.

Structured data ([Self Describing Event](/docs/fundamentals/events/#self-describing-events) payloads and [Entities](/docs/fundamentals/entities/)) will have keys with a prefix of `unstruct_event_` or `contexts_`, the vendor name converted to snake case, the event/entity name in snake case, and the [schema model version](/docs/fundamentals/schemas/versioning/#schema-version-format). For example:

| **Schema URI**                                                  | **Type** | **Key**                                              |
| --------------------------------------------------------------- | -------- | ---------------------------------------------------- |
| `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0` | Entity   | `contexts_com_snowplowanalytics_snowplow_web_page_1` |
| `iglu:org.w3/PerformanceNavigationTiming/jsonschema/1-0-0`      | Entity   | `contexts_org_w3_performance_navigation_timing_1`    |
| `iglu:com.urbanairship.connect/OPEN/jsonschema/1-0-0`           | Event    | `unstruct_event_com_urbanairship_connect_open_1`     |

Timestamps in structured data values will always have their primitive representation (string/number) and not be `time.Time` instances.

## Transforming Data

For all the below examples, the input is a string representation of the below JSON object. For Snowplow data, using `snowplow_mode` will produce a JSON object input - see [the snowplow example](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/).

```json
{

  "name": "Bruce",

  "id": "b47m4n",

  "batmobileCount": 1

}
```

To modify the message data, return an object which conforms to EngineProtocol, with the `Data` field set to the modified data. The `Data` field may be returned as either a string, or an object.

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-modify-example.js&#xA;>)

## Filtering

If the `FilterOut` field of the output is returned as `true`, the message will be acknowledged immediately and won't be sent to the target. This will be the behavior regardless of what is returned to the other fields in the protocol.

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-filter-example.js&#xA;>)

## Setting the Partition Key

To set the Partition Key in the message, you can simply set the input's PartitionKey field, and return it:

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-setpk-example.js&#xA;>)

Or, if modifying the data as well, return the modified data and PartitionKey field:

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-setpk-modify-example.js&#xA;>)

## Setting an HTTP header

For the `http` target only, you can specify a set of HTTP headers, which will be appended to the configured headers for the `http` target. Do so by providing an object in the `HTTPHeaders` field:

```js
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-header-example.js&#xA;>)

The headers will only be included if the target has the [`dynamic_headers = true` setting](/docs/api-reference/snowbridge/configuration/targets/http/#configuration-options) configured.

## Helper functions

- `hash(input, algorithm)` hashes the input value. Salt is configured using the `hash_salt_secret` parameter in the hcl configuration. If no value is provided, this function will perform an unsalted hash

The following hash algorithms are supported:

- `sha1` - SHA-1 hash (160 bits)
- `sha256` - SHA-256 hash (256 bits)
- `md5` - MD5 hash (128 bits)

```js
hash(input.Data["app_id"], "sha1")
```

- `allContexts(data, contextName)` returns an array of all entity instances for a given entity name across all schema versions present in the data object. Instances are ordered by schema version descending (highest version first). `contextName` is the snake\_case vendor and name portion of the key, without the `contexts_` prefix and version suffix. Returns an empty array if no matching keys are found.

> **Note:** `data` must be a parsed object, not a raw string. Use this function with `snowplow_mode`, or after parsing the data with `JSON.parse()`.

1. Access a field from the first entity instance:

```js
// input.Data: {"contexts_nl_basjes_yauaa_context_1":[{"agentName":"python-requests",...}]}



function main(x) {

  const data = JSON.parse(x.Data);

  x.Data = allContexts(data, "nl_basjes_yauaa_context")[0].agentName;

  // returns "python-requests"

  return x;

}
```

2. When multiple schema versions of the same entity are present, concatenate `label` values, with the highest version first:

```js
// input.Data: {"contexts_com_example_1":[{"label":"v1"}],"contexts_com_example_2":[{"label":"v2"}]}



function main(x) {

  const data = JSON.parse(x.Data);

  x.Data = allContexts(data, "com_example").map(c => c.label).join(",");

  // returns "v2,v1"

  return x;

}
```

3. When no matching entity is found, the function returns an empty array — use an optional chain to avoid errors:

```js
// input.Data: {"contexts_com_other_1":[{"url":"https://example.com"}]}



function main(x) {

  const data = JSON.parse(x.Data);

  const instances = allContexts(data, "com_example");

  x.Data = instances[0]?.url ?? "not_found";

  // returns "not_found"

  return x;

}
```

- `allUnstruct(data, eventName)` returns the payload object for a self-describing event matching the given event name. `eventName` is the snake\_case vendor and name portion of the key, without the `unstruct_event_` prefix and version suffix. Returns `undefined` if no matching key is found.

> **Note:** `data` must be a parsed object, not a raw string. Use this function with `snowplow_mode`, or after parsing the data with `JSON.parse()`.

1. Access a field from the event payload:

```js
// input.Data: {"unstruct_event_com_snowplowanalytics_snowplow_add_to_cart_1":{"sku":"item41",...}}



function main(x) {

  const data = JSON.parse(x.Data);

  x.Data = allUnstruct(data, "com_snowplowanalytics_snowplow_add_to_cart").sku;

  // returns "item41"

  return x;

}
```

2. When no matching event is found, the function returns `undefined` — use the nullish coalescing operator (`??`) to provide a fallback:

```js
// input.Data: {"unstruct_event_com_other_1":{"sku":"item1"}}



function main(x) {

  const data = JSON.parse(x.Data);

  x.Data = allUnstruct(data, "com_example") ?? "not_found";

  // returns "not_found"

  return x;

}
```

- `parseTimestamp(timestampStr, layout)` parses a timestamp string into a JavaScript `Date`. The `layout` argument is optional and defaults to `2006-01-02 15:04:05.999`, the format used by Snowplow enriched event timestamp fields such as `collector_tstamp`. To parse a different format, pass a `layout` using [Go reference time syntax](https://pkg.go.dev/time#pkg-constants), where each component of the reference time `Mon Jan 2 15:04:05 MST 2006` stands for that part of the timestamp. The function throws a `RangeError` if the input does not match the layout.

1. Parse a Snowplow timestamp using the default layout:

```js
// input.Data: {"collector_tstamp":"2026-06-15 09:30:00.000"}



function main(x) {

  const data = JSON.parse(x.Data);

  const ts = parseTimestamp(data.collector_tstamp);

  x.Data = ts.toISOString();

  // returns "2026-06-15T09:30:00.000Z"

  return x;

}
```

2. Parse a custom format by passing a `layout`:

```js
// input.Data: {"event_time":"15/06/2026"}



function main(x) {

  const data = JSON.parse(x.Data);

  const ts = parseTimestamp(data.event_time, "02/01/2006");

  x.Data = ts.getUTCFullYear();

  // returns 2026

  return x;

}
```

## Configuration

Once your script is ready, you can configure it in the app by following the [Javascript](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) configuration page.

You can also find some complete example use cases in [the examples section](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/).

---

# Snowbridge JavaScript transformation configuration
> Configure Snowbridge JavaScript transformations using the goja embedded JavaScript engine with script paths and timeout settings.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/

This section details how to configure the transformation, once a script is written.

You can also find some complete example use cases in [the examples section](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/).

The custom JavaScript script transformation uses the [goja](https://pkg.go.dev/github.com/dop251/goja) embedded Javascript engine to run scripts upon the data.

If a script errors or times out, a [transformation failure](/docs/api-reference/snowbridge/concepts/failure-model/#transformation-failure) occurs.

Scripts must be available to the runtime of the application at the path provided in the `script_path` configuration option. For docker, this means mounting the script to the container and providing that path.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/js-configuration-minimal-example.hcl&#xA;>)

Every configuration option:

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/custom-scripts/js-configuration-full-example.hcl&#xA;>)

---

# Snowbridge transformation configuration
> Configure transformations and filters to modify or exclude messages with built-in transformations or custom scripts.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/

You can configure any number of transformations to run on the data one after another - transformations will run in the order provided. (You can repeatedly specify the same transformation more than once, if needed.) All transformations operate on a single message basis.

If you're filtering the data, it's best to provide the filter first, for efficiency.

If you're working with Snowplow enriched messages, you can configure scripting transformations, or any of the built-in transformations, which are specific to Snowplow data.

If you're working with any other type of data, you can create transformations via scripting transformations.

## Transformations and filters

Transformations modify messages in-flight. They might rename fields, perform computations, set partition keys, or modify data. For example if you wanted to change a `snake_case` field name to `camelCase`, you would use a transformation to do this.

Filters are a type of transformation which prevent Snowbridge from further processing data based on a condition. When data is filtered, Snowbridge will ack the message without sending it to the target. For example if you only wanted to send page views to the destination, you would set up a filter with a condition where `event_name` matches the string `page_view`.

## Configuration

Configure transformations by supplying a single `transform {}` block. Within it, choose each transformation using `use "transformation_name" {}`. Multiple transformations run in the order they appear.

Example:

The below first filters out any `event_name` which does not match the regex `^page_view$`, then runs a custom javascript script to change the app\_id value to `"1"`

```hcl
loading...
```

[View on GitHub](<&#xA;https://github.com/snowplow/snowbridge/blob/v5.2.0/assets/docs/configuration/transformations/transformations-overview-example.hcl&#xA;>)

> **Note:** In Snowbridge 4.x, each transformation required its own `transform {}` block. From version 5.0.0, all transformations must be nested inside a single `transform {}` block. See the [upgrade guide](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-5-X-X/) for migration details.

### Worker pool

The `transform {}` block accepts an optional `worker_pool` parameter that controls the number of concurrent goroutines used to run transformations. By default, the worker pool scales automatically based on `GOMAXPROCS`.

```hcl
transform {

  worker_pool = 4



  use "spEnrichedToJson" {}

}
```

Set `worker_pool` explicitly when you need predictable concurrency, for example in memory-constrained environments.

---

# Getting started with Snowbridge
> Install and configure Snowbridge using Docker or binaries to start replicating event streams between sources and targets.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/getting-started/

The fastest way to get started and experiment with Snowbridge is to run it via the command line:

1. Download the pre-compiled ZIP from the [releases](https://github.com/snowplow/snowbridge/releases/)
2. Unzip and run the binary with eg. `echo "hello world" | ./snowbridge`

The defaults for the app are stdin source, no transformations, and stdout target - so this should print the message 'hello world' along with some logging data to the console.

Next, the app can be configured using HCL - simply create a configuration file, and provide the path to it using the `SNOWBRIDGE_CONFIG_FILE` environment variable.

You can find a guide to configuration in the [configuration section](/docs/api-reference/snowbridge/configuration/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Snowbridge (since version 1.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `user_provided_id` configuration setting.

If you wish to disable telemetry, you can do so by setting `disable_telemetry` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

## Distribution options

There are two distributions of Snowbridge.

**Default:**

The default distribution contains everything except for the [Kinesis source](/docs/api-reference/snowbridge/configuration/sources/kinesis/), i.e. the ability to read from AWS Kinesis. This distribution is all licensed under the [Snowplow Limited Use License Agreement](/limited-use-license-1.0/). _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions](/docs/licensing/limited-use-license-faq/).)_

It’s available on Docker:

```bash
docker pull snowplow/snowbridge:5.2.0

docker run snowplow/snowbridge:5.2.0
```

**AWS-specific (includes Kinesis source):**

The AWS-specific distribution contains everything, including the [Kinesis source](/docs/api-reference/snowbridge/configuration/sources/kinesis/), i.e. the ability to read from AWS Kinesis. Like the default distribution, it’s licensed under the [Snowplow Limited Use License Agreement](/limited-use-license-1.0/) ([frequently asked questions](/docs/licensing/limited-use-license-faq/)). However, this distribution has a dependency on [twitchscience/kinsumer](https://github.com/twitchscience/kinsumer), which is licensed by Twitch under the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE).

To comply with the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE), you may only use this distribution of Snowbridge _“with the web services, computing platforms or applications provided by Amazon.com, Inc. or its affiliates, including Amazon Web Services, Inc.”_

It’s available on Docker:

```bash
docker pull snowplow/snowbridge:5.2.0-aws-only

docker run snowplow/snowbridge:5.2.0-aws-only
```

***

## Deployment

The app can be deployed via services like EC2, ECS or Kubernetes using docker.

Configuration and authentication can be done by mounting the relevant files, and/or setting the relevant environment variables as per the standard authentication methods for cloud services.

---

# Replicate event streams in real time with Snowbridge
> Replicate Snowplow event streams to multiple destinations with Snowbridge, a configurable tool supporting Kinesis, PubSub, Kafka, HTTP, and more.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/

Snowbridge is a flexible, low latency tool which can replicate streams of data of any type to external destinations, optionally filtering or transforming the data along the way. It can be used to consume, transform and relay data to any third party platform which supports HTTP or is listed as a target below — in real-time.

## Features

- [Kinesis](https://aws.amazon.com/kinesis), [SQS](https://aws.amazon.com/sqs/), [PubSub](https://cloud.google.com/pubsub), [Kafka](https://kafka.apache.org/), and stdin sources

- [Kinesis](https://aws.amazon.com/kinesis), [SQS](https://aws.amazon.com/sqs/), [PubSub](https://cloud.google.com/pubsub), [Kafka](https://kafka.apache.org/), [Event Hubs](https://azure.microsoft.com/en-us/services/event-hubs/), HTTP (e.g. for an [integration with Google Tag Manager Server Side](/docs/destinations/forwarding-events/google-tag-manager-server-side/)), and stdout targets

- Custom in-flight JS transformations

- Low-latency Snowplow-specific data transformations

- Statsd and Sentry reporting and monitoring interfaces

Snowbridge is a generic tool, built to work on any type of data, developed by the Snowplow team. It began life as a closed-source tool developed to deliver various requirements related to Snowplow data, and so some of the features are specific to that data.

> **Note:** Version 4.0.0 includes breaking changes to response rule evaluation. See the [upgrade guide](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-4-X-X/) for migration information.

> **Note:** Version 5.0.0 includes breaking changes to transformation configuration and target batching options. See the [upgrade guide](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-5-X-X/) for migration information.

---

# Testing Snowbridge locally
> Test Snowbridge configurations locally using stdin and stdout sources before deploying to production stream infrastructure.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/testing/

The easiest way to test Snowbridge configuration (e.g. transformations) is to run it locally. Ideally, you should also use a sample of data that is as close to your real world data as possible. The sample file should contain the events/messages you’d like to test with, one per line.

## Snowplow data

You can get started working with Snowplow data by [downloading this file](/assets/files/input-5ef65c98e57e93e452e9f6bd5a413fed.txt/) which contains a sample of web and mobile Snowplow events in TSV format. However, if you need events that match your actual events, generate your own events sample.

In order to generate your own sample of Snowplow data, you can follow the [guide to use Snowplow Micro](/docs/testing/snowplow-micro/local/) to generate test data, using the `--output-tsv` to get the data into a file, as per the [exporting to tsv section](/docs/testing/snowplow-micro/local/#exporting-events).

For example, here we’re using a file named `data.tsv`:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.4.0 --output-tsv > data.tsv
```

Point some test environment tracking to `localhost:9090`, and your events should land in `data.tsv`.

## Testing Snowbridge locally

You can run Snowbridge locally via Docker:

```bash
docker run --env ACCEPT_LIMITED_USE_LICENSE=yes snowplow/snowbridge:5.2.0
```

The default configuration for Snowbridge uses the `stdin` source and the `stdout` target. So, to test sending data through with no transformations, we can run the following command (where `data.tsv` is a file with Snowplow events in TSV format):

```bash
cat data.tsv | docker run --env ACCEPT_LIMITED_USE_LICENSE=yes -i snowplow/snowbridge:5.2.0
```

This will print the data to the terminal, along with logs.

> **Note:** The metrics reported in the logs may state that no data has been processed. This is because the app reached the end of output and exited before the default reporting period of 1 second. You can safely ignore this.

You can output the results to a file to make it easier to examine them:

```bash
cat data.tsv | docker run -i snowplow/snowbridge:5.2.0 > output.txt
```

> **Tip:** The output (in `output.txt`) will contain more than the data itself. There will be additional fields called `PartitionKey`, `TimeCreated` `TimePulled` and `TimeTransformed`. The data that reaches the target in a production setup is under `Data`.

### Adding configuration

To add a specific configuration to test, create a configuration file (`config.hcl`) and pass it to the Docker container. You will need to [mount the file](https://docs.docker.com/storage/bind-mounts/) to the default config path of `/tmp/config.hcl`:

```bash
cat data.tsv | docker run -i \

  --env ACCEPT_LIMITED_USE_LICENSE=yes \

  --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \

  snowplow/snowbridge:5.2.0 > output.txt
```

Note that docker expects absolute paths for mounted files - here we use `$(pwd)` but you can specify the aboslute path manually too.

To test transformations, you only need to add the `transform` block(s) to your configuration file. Don’t specify the `source` and `target` blocks to leave them on default (`stdin` and `stdout`).

To test specific sources or targets, add the respective `source` or `target` blocks. For example, see the [configuration](/docs/api-reference/snowbridge/configuration/targets/http/#example-for-google-tag-manager-server-side) for an HTTP target sending data to Google Tag Manager Server Side.

### Adding a custom transformation script

You can add custom scripts by mounting a file, similarly to the above. Assuming the script is in `script.js`, that looks like this:

```bash
cat data.tsv | docker run -i \

  --env ACCEPT_LIMITED_USE_LICENSE=yes \

  --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \

  --mount type=bind,source=$(pwd)/script.js,target=/tmp/script.js \

  snowplow/snowbridge:5.2.0 > output.txt
```

The transformation config should point to the path of the script _inside_ the container (`/tmp/script.js` above). For example, the transformation block in the configuration might look like this:

```hcl
transform {

  use "js" {

    script_path = "/tmp/script.js"

  }

}
```

### Adding an HTTP request template

For the HTTP target, you can add a custom HTTP request template by mounting a file, similarly to above. Assuming the template is in `template.tpl`, that looks like this:

```bash
cat data.tsv | docker run -i \

  --env ACCEPT_LIMITED_USE_LICENSE=yes \

  --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \

  --mount type=bind,source=$(pwd)/template.tpl,target=/tmp/template.tpl \

  snowplow/snowbridge:5.2.0
```

The HTTP target config should point to the path of the script _inside_ the container (`/tmp/template.tpl` above). For example, the HTTP target block in the configuration might look like this:

```hcl
target {

  use "http" {

    url             = "http://myApi.com/events"

    template_file   = "/tmp/template.tpl"

  }

}
```

## Using Docker Compose

Instead of creating an intermediate output file with Snowplow Micro and then processing that file in Snowbridge, you can also use Docker Compose to run Micro and Snowbridge together (since Micro 3.0.0 and Snowbridge 3.6.2).

This way, any events sent to Micro will immediately make it to Snowbridge.

Here is an example setup. Copy the code into `docker-compose.yml` and run with `docker-compose up`:

```yaml
services:

  micro:

    image: snowplow/snowplow-micro:4.4.0

    ports:

      - "9090:9090"

    command: --output-tsv --destination http://snowbridge:8080

  snowbridge:

    image: snowplow/snowbridge:5.2.0

    environment:

      - SNOWBRIDGE_CONFIG_FILE=/tmp/config.hcl

    configs:

      - source: snowbridge_config

        target: /tmp/config.hcl



configs:

  snowbridge_config:

    content: |

      license {

        accept = true

      }



      source {

        use "http" {

          url = "0.0.0.0:8080"

        }

      }



      transform {

        use "spEnrichedToJson" {

        }

      }



      # any other Snowbridge configuration
```

## Further testing

You can use either method above (TSV file or Docker Compose) to test all aspects of the app from a local environment too, including sources, targets, failure targets, metrics endpoints etc. In some cases, you'll need to ensure that the local environment has access to any required resources and can authenticate, such as connecting from a laptop to a cloud account/local mock of cloud resources, or setting up a local metrics server for testing. Once that’s done, provide Snowbridge with an hcl file configuring it to connect to those resources, and run it the same way as in the examples above.

---

# Upgrade guides for Snowbridge
> Step-by-step guides for upgrading Snowplow Snowbridge to newer versions with breaking changes and migration instructions.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/upgrade-guides/

This section contains information to help you upgrade to newer versions of Snowbridge.

## [📄️Snowbridge 3.x upgrade](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-3-X-X/)

[Upgrade Snowbridge to version 3.X.X with configuration changes for transformations, new features, and breaking changes.](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-3-X-X/)

## [📄️Snowbridge 4.x upgrade](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-4-X-X/)

[Upgrade Snowbridge to version 4.X.X with configuration changes for transformations, new features, and breaking changes.](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-4-X-X/)

## [📄️Snowbridge 5.x upgrade](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-5-X-X/)

[Upgrade Snowbridge to version 5.X.X with breaking changes to transformation configuration, target batching, and metrics.](/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-5-X-X/)

---

# Snowbridge 3.x upgrade guide
> Upgrade Snowbridge to version 3.X.X with configuration changes for transformations, new features, and breaking changes.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-3-X-X/

## Version 3.0.0 Breaking Changes

The below breaking changes were made in version 3.0.0. All other functionality is backwards compatible.

### Lua support removed

Support for Lua transformations has been removed. If you are running a Lua transformation, you can port the logic to [Javascript](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) or [JQ](/docs/api-reference/snowbridge/configuration/transformations/builtin/jq/).

### HTTP target: non-JSON data no longer supported

We never intended to support non-JSON data, but prior to version 3.0.0, the request body was simply populated with whatever bytes were found in the message data, regardless of whether it is valid JSON.

From version 3.0.0 onwards, only valid JSON will work, otherwise the message will be considered invalid and sent to the failure target.

### HTTP target: request batching

Many HTTP APIs allow sending several events in a single request by putting them into a JSON array. Since version 3.0.0, if the Snowbridge source provides data in batches, the HTTP target will batch events in this way.

As a consequence, even when the source provides events in a single event batch, it will now be placed into an array of one element. For example, prior to version 3.0.0, a request body might look like this:

```text
{"foo": "bar"}
```

But it will now look like this:

```text
[{"foo": "bar"}]
```

As of version 3.0.0, the SQS source provides events in batches of up to ten, and the Kinesis, Kafka, and Pubsub and Stdin sources provide events in single-event batches. This behavior will likely change in a future version.

You can preserve the previous behavior and ensure that requests are always single-event non-array objects, even with a batching source. To do so, set `request_max_messages` to 1, and provide this template (as long as your data is valid JSON):

```go
loading...
```

[View on GitHub](<https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/targets/http-template-unwrap-example.file&#xA;>)

---

# Snowbridge 4.x upgrade guide
> Upgrade Snowbridge to version 4.X.X with configuration changes for transformations, new features, and breaking changes.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-4-X-X/

## Version 4.0.0 Breaking Changes

### HTTP target: ordered response rule evaluation

**Breaking change**: response rules are now evaluated in the order they are defined in the configuration, rather than being organized in separate `invalid` and `setup` blocks.

**Migration required**: you must update your HTTP target configuration to specify a `type` attribute for each rule:

**Before:**

```hcl
response_rules {

  invalid {

    http_codes = [400]

    body = "Invalid value for 'purchase' field"

  }

  setup {

    http_codes = [401, 403]

  }

}
```

**After (4.0.0):**

```hcl
response_rules {

  rule {

    type = "invalid"

    http_codes = [400]

    body = "Invalid value for 'purchase' field"

  }

  rule {

    type = "setup"

    http_codes = [401, 403]

  }

}
```

**Important**: rules are now evaluated in the order they appear in your configuration. The first matching rule determines the error type.

---

# Snowbridge 5.x upgrade guide
> Upgrade Snowbridge to version 5.X.X with breaking changes to transformation configuration, target batching, and metrics.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/upgrade-guides/upgrade-guide-5-X-X/

## Version 5.0.0 breaking changes

### All sources

`concurrent_writes` setting has been removed. Concurrency is now configured at transformation and target levels: see [batching model](/docs/api-reference/snowbridge/concepts/batching-model/) for more details.

### Transformation configuration syntax

Multiple `transform {}` blocks are replaced by a single `transform {}` block containing multiple `use` sub-blocks.

**Migration required**: consolidate your transformation blocks into one.

**Before:**

```hcl
transform {

  use "spEnrichedFilter" {

    atomic_field  = "event_name"

    regex         = "^page_view$"

    filter_action = "keep"

  }

}



transform {

  use "js" {

    script_path = "/opt/script.js"

  }

}
```

**After (5.0.0):**

```hcl
transform {

  use "spEnrichedFilter" {

    atomic_field  = "event_name"

    regex         = "^page_view$"

    filter_action = "keep"

  }



  use "js" {

    script_path = "/opt/script.js"

  }

}
```

Transformations still run in the order they appear.

### HTTP target

#### Headers configuration

`headers` setting now accepts an object structure to define custom headers.

**Before:**

```hcl
target {

  use "http" {

    ...

    headers  = "{\"Accept-Language\":\"en-US\"}"

  }

}
```

**After (5.0.0):**

```hcl
target {

  use "http" {

    ...

    headers = {

      Accept-Language = "en-US"

    }

  }

}
```

#### Removal of deprecated settings

`request_timeout_in_seconds` setting has been removed.

**Migration required**: use `request_timeout_in_millis` setting instead.

#### Batching options renamed and moved

The per-request sizing options on the HTTP target have been renamed and moved into a nested `batching {}` block.

| v4 field (top-level)   | v5 field (inside `batching {}`) | Default   |
| ---------------------- | ------------------------------- | --------- |
| `request_max_messages` | `max_batch_messages`            | `50`      |
| `request_byte_limit`   | `max_batch_bytes`               | `1048576` |
| `message_byte_limit`   | `max_message_bytes`             | `1048576` |

Two additional fields are new in v5:

| New field                | Default | Description                                           |
| ------------------------ | ------- | ----------------------------------------------------- |
| `max_concurrent_batches` | `5`     | Number of batches written to the target concurrently. |
| `flush_period_millis`    | `500`   | Milliseconds between timer-based batch flushes.       |

**Before:**

```hcl
target {

  use "http" {

    url                  = "https://acme.com/x"

    request_max_messages = 20

    request_byte_limit   = 10000000

    message_byte_limit   = 10000000

  }

}
```

**After (5.0.0):**

```hcl
target {

  use "http" {

    url = "https://acme.com/x"



    batching {

      max_batch_messages     = 20

      max_batch_bytes        = 10000000

      max_message_bytes      = 10000000

      max_concurrent_batches = 5

      flush_period_millis    = 500

    }

  }

}
```

### EventHub target

Fields have been renamed and moved into a nested `batching {}` block:

| v4 field (top-level)  | v5 field (inside `batching {}`) | Default   |
| --------------------- | ------------------------------- | --------- |
| `chunk_message_limit` | `max_batch_messages`            | `500`     |
| `chunk_byte_limit`    | `max_batch_bytes`               | `1048576` |
| `message_byte_limit`  | `max_message_bytes`             | `1048576` |
| `batch_byte_limit`    | `max_batch_bytes`               | `1048576` |

Two additional fields are new in v5:

| New field                | Default | Description                                           |
| ------------------------ | ------- | ----------------------------------------------------- |
| `max_concurrent_batches` | `5`     | Number of batches written to the target concurrently. |
| `flush_period_millis`    | `500`   | Milliseconds between timer-based batch flushes.       |

**Before:**

```hcl
target {

  use "eventhub" {

    message_byte_limit         = 1000000

    chunk_byte_limit           = 1000000

    chunk_message_limit        = 501

    batch_byte_limit           = 1000000

  }

}
```

**After (5.0.0):**

```hcl
target {

  use "eventhub" {

    batching {

      max_batch_messages     = 501

      max_batch_bytes        = 10000000

      max_message_bytes      = 10000000

      max_concurrent_batches = 5

      flush_period_millis    = 500

    }

  }

}
```

### Kafka target

Fields have been renamed and moved into a nested `batching {}` block:

| v4 field (top-level) | v5 field (inside `batching {}`) | Default   |
| -------------------- | ------------------------------- | --------- |
| `flush_messages`     | `max_batch_messages`            | `100`     |
| `flush_bytes`        | `max_batch_bytes`               | `1048576` |
| `byte_limit`         | `max_message_bytes`             | `1048576` |
| `flush_frequency`    | `flush_period_millis`           | `500`     |

One additional field is new in v5:

| New field                | Default | Description                                           |
| ------------------------ | ------- | ----------------------------------------------------- |
| `max_concurrent_batches` | `5`     | Number of batches written to the target concurrently. |

**Before:**

```hcl
target {

  use "kafka" {

    byte_limit      = 1000000

    flush_frequency = 2

    flush_messages  = 2

    flush_bytes     = 2

  }

}
```

**After (5.0.0):**

```hcl
target {

  use "kafka" {

    batching {

      max_batch_messages     = 501

      max_batch_bytes        = 10000000

      max_message_bytes      = 10000000

      flush_period_millis    = 500

      max_concurrent_batches = 5

    }

  }

}
```

### Kinesis target

Field have been renamed and moved into a nested `batching {}` block:

| v4 field (top-level)   | v5 field (inside `batching {}`) | Default |
| ---------------------- | ------------------------------- | ------- |
| `request_max_messages` | `max_batch_messages`            | `500`   |

Four additional fields are new in v5:

| New field                | Default   | Description                                           |
| ------------------------ | --------- | ----------------------------------------------------- |
| `max_batch_bytes`        | `5242880` | Maximum byte limit for a single batched request.      |
| `max_message_bytes`      | `1048576` | Maximum byte limit for individual message.            |
| `max_concurrent_batches` | `5`       | Number of batches written to the target concurrently. |
| `flush_period_millis`    | `500`     | Milliseconds between timer-based batch flushes.       |

**Before:**

```hcl
target {

  use "kinesis" {

    request_max_messages = 1

  }

}
```

**After (5.0.0):**

```hcl
target {

  use "kinesis" {

    batching {

      max_batch_messages     = 1

      max_batch_bytes        = 5242880

      max_message_bytes      = 1048576

      max_concurrent_batches = 5

      flush_period_millis    = 500

    }

  }

}
```

### PubSub & SQS targets

Similarly to the rest of the targets, `PubSub` and `SQS` targets now support a nested `batching {}` block:

- Pubsub

| Field                    | Default    | Description                                                   |
| ------------------------ | ---------- | ------------------------------------------------------------- |
| `max_batch_messages`     | `100`      | Maximum number of events that can go into one batched request |
| `max_batch_bytes`        | `10485760` | Maximum byte limit for a single batched request.              |
| `max_message_bytes`      | `10485760` | Maximum byte limit for individual message.                    |
| `max_concurrent_batches` | `5`        | Number of batches written to the target concurrently.         |
| `flush_period_millis`    | `500`      | Milliseconds between timer-based batch flushes.               |

- SQS

| Field                    | Default   | Description                                                   |
| ------------------------ | --------- | ------------------------------------------------------------- |
| `max_batch_messages`     | `10`      | Maximum number of events that can go into one batched request |
| `max_batch_bytes`        | `1048576` | Maximum byte limit for a single batched request.              |
| `max_message_bytes`      | `1048576` | Maximum byte limit for individual message.                    |
| `max_concurrent_batches` | `5`       | Number of batches written to the target concurrently.         |
| `flush_period_millis`    | `500`     | Milliseconds between timer-based batch flushes.               |

### All targets

If `batching {}` block is not specified for the selected target, then target would use it's default batching configuration.

---

# Manage data structures via Snowplow CLI
> Use the Snowplow CLI data-structures command to generate, download, validate, and publish data structures with git-ops workflows and JSON Schema validation.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-cli/data-structures/

The `data-structures` subcommand of [Snowplow CLI](/docs/api-reference/snowplow-cli/) provides a collection of functionality to ease the integration of custom development and publishing workflows.

## Snowplow CLI Prerequisites

Installed and configured [Snowplow CLI](/docs/api-reference/snowplow-cli/)

## Available commands

### Creating data structures

```bash
snowplow-cli ds generate login_click ./folder-name
```

Will create a minimal data structure template in a new file `./folder-name/login_click.yaml`. Note that you will need to add a vendor name to the template before it will pass validation. Alternatively supply a vendor at creation time with the `--vendor com.acme` flag.

### Downloading data structures

```bash
snowplow-cli ds download
```

This command will retrieve all organization data structures. By default it will create a folder named `data-structures` in the current working directory to put them in. It uses a combination of vendor and name to further break things down.

Given a data structure with `vendor: com.acme` and `name: link_click` and assuming the default format of yaml the resulting folder structure will be `./data-structures/com.acme/link_click.yaml`.

> **Note:** The CLI download command only retrieves data structures that have been deployed to at least the development environment. **Draft data structures** that haven't been deployed yet will not be included in the download.

### Validating data structures

```bash
snowplow-cli ds validate ./folder-name
```

This command will find all files under `./folder-name` (if omitted then `./data-structures`) and attempt to validate them using Snowplow Console. It will assert the following

1. Is each file a valid format (yaml/json) with expected fields
2. Does the schema in the file conform to [snowplow expectations](/docs/fundamentals/schemas/#self-describing-json-schema-anatomy)
3. Given the organization's [loading configuration](/docs/destinations/warehouses-lakes/) will any schema version number choices have a potentially negative effect on data loading

If any validations fail the command will report the problems to stdout and exit with status code 1.

### Publishing data structures

```bash
snowplow-cli ds publish dev ./folder-name
```

This command will find all files under `./folder-name` (if omitted then `./data-structures`) and attempt to publish them to Snowplow Console in the environment provided (`dev` or `prod`).

Publishing to `dev` will also cause data structures to be validated with the `validate` command before upload. Publishing to `prod` will not validate but requires all data structures referenced to be present on `dev`.

---

# Snowplow CLI for data management
> Command-line tool for downloading and syncing data structures and tracking plans to Snowplow Console, enabling git-ops workflows with reviews and branching.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-cli/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

Snowplow CLI brings data management elements of Snowplow Console into the command line. It allows you to download your data structures and tracking plans to YAML/JSON files and publish them back to Console. This enables git-ops-like workflows, where your tracking design lives in your repository alongside your application code. Changes are reviewed and deployed through your standard development process.

## Install

Snowplow CLI can be installed with [homebrew](https://brew.sh/):

```bash
brew install snowplow/taps/snowplow-cli
```

Verify the installation with

```bash
snowplow-cli --help
```

For systems where homebrew is not available binaries for multiple platforms can be found in [releases](https://github.com/snowplow/snowplow-cli/releases).

Example installation for `linux_x86_64` using `curl`

```bash
curl -L -o snowplow-cli https://github.com/snowplow/snowplow-cli/releases/latest/download/snowplow-cli_linux_x86_64

chmod u+x snowplow-cli
```

Verify the installation with

```bash
snowplow-cli --help
```

## Configure

### Automated Setup

The easiest way to configure Snowplow CLI is using the built-in `setup` command:

```bash
snowplow-cli setup
```

This command will:

- Guide you through device authentication with Snowplow Console
- Automatically create and configure your API credentials
- Set up your organization ID

**Prerequisites:** Your Snowplow Console account must have sufficient permissions to create API keys.

You can also use optional flags:

- `--read-only`: Create a read-only API key
- `--dotenv`: Store configuration as .env file in current working directory

> **Note:** If SSO is enabled for your Snowplow organization, you will need to follow the [Manual Configuration](#manual-configuration) method instead.

### Manual Configuration

If you prefer manual configuration, you will need these values:

- An API Key ID and the corresponding API Key (secret), which are generated from the [credentials section](https://console.snowplowanalytics.com/credentials) in Console.
- Your Organization ID, which you can find [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

Snowplow CLI can take its configuration from a variety of sources. More details are available from `snowplow-cli data-structures --help`. Variations on these three examples should serve most cases.

**env variables or .dotenv file:**

```bash
SNOWPLOW_CONSOLE_API_KEY_ID=********-****-****-****-************

SNOWPLOW_CONSOLE_API_KEY=********-****-****-****-************

SNOWPLOW_CONSOLE_ORG_ID=********-****-****-****-************
```

**$HOME/.config/snowplow/snowplow\.yml:**

```yaml
console:

  api-key-id: ********-****-****-****-************

  api-key: ********-****-****-****-************

  org-id: ********-****-****-****-************
```

**inline arguments:**

```bash
snowplow-cli data-structures --api-key-id ********-****-****-****-************ --api-key ********-****-****-****-************ --org-id ********-****-****-****-************
```

***

Snowplow CLI defaults to yaml format. It can be changed to json by either providing a `--output-format json` flag or setting the `output-format: json` config value. It will work for all commands where it matters, not only for `generate`.

### Verify Configuration

After configuration, you can verify that everything is working correctly using the `status` command:

```bash
snowplow-cli status
```

This command will:

- Check that your API credentials are properly configured
- Verify connectivity to Snowplow Console
- Confirm your organization access
- Provide helpful troubleshooting information if issues are found

If the status check fails, the command will suggest the next steps such as running `snowplow-cli setup` to reconfigure your credentials.

### Configure YAML language server (optional)

During the tracking plans authoring process it's convenient to have the editor highlight errors, suggest possible values, and show examples. All files generated by Snowplow CLI's `generate` and `download` commands will contain a special comment as part of the file. This comment can be interpreted by a YAML Language server.

For VS Code you can get this authoring functionality using this [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml). Instructions for other editors can be found [here](https://github.com/redhat-developer/yaml-language-server?tab=readme-ov-file#clients).

After that running `snowplow-cli ds generate test` and opening the generated file in the configured editor of choice should look something like the following:

![VS Code editor showing a Snowplow data structure YAML file with YAML language server validation active, highlighting a type error on the vendor field with the message \&quot;Incorrect type. Expected \&#39;string\&#39;\&quot;](/assets/images/lspValidation-21bfbfb84fe78d4b98d42b0ef09eda5b.png)

## MCP server

The Snowplow CLI includes a local Model Context Protocol (MCP) server that enables natural language interaction with AI assistants like Claude Desktop, Cursor, GitHub Copilot, and other MCP-compatible clients for creating, validating, and managing your Snowplow tracking plans. This allows you to:

- Create and validate data structures through conversation
- Analyze tracking requirements and suggest implementations
- Validate tracking plans and source applications

For setup instructions and configuration examples for different MCP clients, see our [MCP tutorial](/tutorials/snowplow-cli-mcp/introduction/).

## Use cases

- [Manage your data structures with snowplow-cli](/docs/api-reference/snowplow-cli/data-structures/)
- [Send events from the command line with snowplow-cli](/docs/api-reference/snowplow-cli/send-events/)
- [Set up a GitHub CI/CD pipeline to manage data structures and tracking plans](/tutorials/data-structures-in-git/introduction/)

---

# Snowplow CLI command reference
> Complete reference for Snowplow CLI commands including data-products and data-structures subcommands with options and usage examples.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-cli/reference/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

This page contains the complete reference for the Snowplow CLI commands.

## Data-Products

Work with Snowplow tracking plans

### Examples

```text
  $ snowplow-cli data-products validate
```

### Options

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

  -h, --help                  help for data-products

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                          Then on:

                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                            Windows %AppData%\snowplow\snowplow.yml

      --debug             Log output level to Debug

      --env-file string   Environment file (.env). Defaults to .env in current directory

                          Then on:

                            Unix $HOME/.config/snowplow/.env

                            Darwin $HOME/Library/Application Support/snowplow/.env

                            Windows %AppData%\snowplow\.env

      --json-output       Log output as json

  -q, --quiet             Log output level to Warn

  -s, --silent            Disable output
```

## Data-Products Add-Event-Spec

Add new event spec to an existing tracking plan

### Synopsis

Adds one or more event specifications to an existing tracking plan file, or print them to the stdout. The command takes the path to a tracking plan file and adds the specified event specifications to it. When run without the argument - it will print generated event specs to the stdout. The command will attempt to keep the formatting and comments of the original file intact, but it's a best effort approach. Some comments might be deleted, some formatting changes might occur.

```text
snowplow-cli data-products add-event-spec {path} [flags]
```

### Examples

```text
  $ snowplow-cli dp add-event-spec --event-spec user_login --event-spec page_view ./my-data-product.yaml

  $ snowplow-cli dp add-es ./data-products/analytics.yaml -e "checkout_completed" -e "item_purchased"
```

### Options

```text
  -e, --event-spec stringArray   Name of event spec to add

  -h, --help                     help for add-event-spec

  -f, --output-format string     Format of stdout output. Only applicable when the file in not specified. Json or yaml are supported (default "yaml")
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Products Download

Download all tracking plans, event specs and source apps from Snowplow Console

### Synopsis

Downloads the latest versions of all tracking plans, event specs and source apps from Snowplow Console.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products download {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp download

  $ snowplow-cli dp download ./my-data-products
```

### Options

```text
  -h, --help                   help for download

  -f, --output-format string   Format of the files to read/write. json or yaml are supported (default "yaml")

      --plain                  Don't include any comments in yaml files
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Products Generate

Generate new tracking plans and source applications locally

### Synopsis

Will write new tracking plans and/or source application to file based on the arguments provided.

Example: $ snowplow-cli dp gen --source-app "Mobile app" Will result in a new source application getting written to './data-products/source-applications/mobile-app.yaml'

$ snowplow-cli dp gen --data-product "Ad tracking" --output-format json --data-products-directory dir1 Will result in a new tracking plan getting written to './dir1/ad-tracking.json'

```text
snowplow-cli data-products generate [paths...] [flags]
```

### Examples

```text
  $ snowplow-cli dp generate --source-app "Mobile app" --source-app "Web app" --data-product "Signup flow"
```

### Options

```text
      --data-product stringArray         Name of tracking plan to generate

      --data-products-directory string   Directory to write tracking plans to (default "data-products")

  -h, --help                             help for generate

      --output-format string             File format (yaml|json) (default "yaml")

      --plain                            Don't include any comments in yaml files

      --source-app stringArray           Name of source app to generate

      --source-apps-directory string     Directory to write source apps to (default "data-products/source-apps")
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Products Purge

Purges (permanently removes) all remote tracking plans and source apps that do not exist locally

### Synopsis

Purges (permanently removes) all remote tracking plans and source apps that do not exist locally.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products purge {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp purge

  $ snowplow-cli dp purge ./my-data-products
```

### Options

```text
  -h, --help   help for purge

  -y, --yes    commit to purge
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Products Release

Sync tracking plans, event specs and source apps to Snowplow Console, then release event specs to be available within your pipeline

### Synopsis

Sync tracking plans, event specs and source apps to Snowplow Console, then release event specs.

This command runs 'sync' first, then releases the event specs. Releasing sets the event spec status to 'published' and pushes them to the pipeline, enabling event spec inference. Only event specs that exist locally will be released. Each event spec must have an event defined, and all referenced events and entities must be published to the production environment.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products release {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp release

  $ snowplow-cli dp release ./my-data-products
```

### Options

```text
  -c, --concurrency int   The number of validation requests to perform at once (maximum 10) (default 3)

  -d, --dry-run           Only print planned changes without performing them

      --gh-annotate       Output suitable for github workflow annotation (ignores -s)

  -h, --help              help for release
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Products Sync

Sync tracking plans, event specs and source apps to Snowplow Console

### Synopsis

Sync tracking plans, event specs and source apps to Snowplow Console.

This command syncs local files with Snowplow Console. Tracking plans, event specs and source apps are created or updated as needed. Tracking plans and source apps that exist in Snowplow Console are updated in place. Structural changes to event specs (name, event, entities) will instead create a new draft version of the event spec. Use 'release' to also release event specs, which changes the status in Snowplow Console to "published" and enables event spec inference.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products sync {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp sync

  $ snowplow-cli dp sync ./my-data-products
```

### Options

```text
  -c, --concurrency int   The number of validation requests to perform at once (maximum 10) (default 3)

  -d, --dry-run           Only print planned changes without performing them

      --gh-annotate       Output suitable for github workflow annotation (ignores -s)

  -h, --help              help for sync
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Products Validate

Validate tracking plans and source applications with Snowplow Console

### Synopsis

Sends all tracking plans and source applications from \<path> for validation by Snowplow Console.

```text
snowplow-cli data-products validate [paths...] [flags]
```

### Examples

```text
  $ snowplow-cli dp validate ./data-products ./source-applications

  $ snowplow-cli dp validate ./src
```

### Options

```text
  -c, --concurrency int   The number of validation requests to perform at once (maximum 10) (default 3)

      --full              Perform compatibility check on all files, not only the ones that were changed

      --gh-annotate       Output suitable for github workflow annotation (ignores -s)

  -h, --help              help for validate
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Structures

Work with Snowplow data structures

### Examples

```text
  $ snowplow-cli data-structures generate my_new_data_structure

  $ snowplow-cli ds validate

  $ snowplow-cli ds publish dev
```

### Options

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

  -h, --help                  help for data-structures

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                          Then on:

                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                            Windows %AppData%\snowplow\snowplow.yml

      --debug             Log output level to Debug

      --env-file string   Environment file (.env). Defaults to .env in current directory

                          Then on:

                            Unix $HOME/.config/snowplow/.env

                            Darwin $HOME/Library/Application Support/snowplow/.env

                            Windows %AppData%\snowplow\.env

      --json-output       Log output as json

  -q, --quiet             Log output level to Warn

  -s, --silent            Disable output
```

## Data-Structures Download

Download all data structures from Snowplow Console

### Synopsis

Downloads the latest versions of all data structures from Snowplow Console.

Will retrieve schema contents from your development environment. If no directory is provided then defaults to 'data-structures' in the current directory.

By default, data structures with empty schemaType (legacy format) are skipped. Use --include-legacy to include them (they will be set to 'entity' schemaType).

```text
snowplow-cli data-structures download {directory ./data-structures} [flags]
```

### Examples

```text
  $ snowplow-cli ds download



  Download data structures matching com.example/event_name* or com.example.subdomain*

  $ snowplow-cli ds download --match com.example/event_name --match com.example.subdomain



  Download with custom output format and directory

  $ snowplow-cli ds download --output-format json ./my-data-structures



  Include legacy data structures with empty schemaType

  $ snowplow-cli ds download --include-legacy
```

### Options

```text
  -h, --help                   help for download

      --include-drafts         Include drafts data structures

      --include-legacy         Include legacy data structures with empty schemaType (will be set to 'entity')

      --match stringArray      Match for specific data structure to download (eg. --match com.example/event_name or --match com.example)

  -f, --output-format string   Format of the files to read/write. json or yaml are supported (default "yaml")

      --plain                  Don't include any comments in yaml files
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Structures Generate

Generate a new data structure locally

### Synopsis

Will write a new data structure to file based on the arguments provided.

Example: $ snowplow-cli ds gen login\_click --vendor com.example Will result in a new data structure getting written to './data-structures/com.example/login\_click.yaml' The directory 'com.example' will be created automatically.

$ snowplow-cli ds gen login\_click Will result in a new data structure getting written to './data-structures/login\_click.yaml' with an empty vendor field. Note that vendor is a required field and will cause a validation error if not completed.

```text
snowplow-cli data-structures generate login_click {directory ./data-structures} [flags]
```

### Examples

```text
  $ snowplow-cli ds generate my-ds

  $ snowplow-cli ds generate my-ds ./my-data-structures
```

### Options

```text
      --entity                 Generate data structure as an entity

      --event                  Generate data structure as an event (default true)

  -h, --help                   help for generate

      --output-format string   Format for the file (yaml|json) (default "yaml")

      --plain                  Don't include any comments in yaml files

      --vendor string          A vendor for the data structure.

                               Must conform to the regex pattern [a-zA-Z0-9-_.]+
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Structures Publish

Publishing commands for data structures

### Synopsis

Publishing commands for data structures

Publish local data structures to Snowplow Console.

### Options

```text
  -h, --help   help for publish
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Structures Publish Dev

Publish data structures to your development environment

### Synopsis

Publish modified data structures to Snowplow Console and your development environment

The 'meta' section of a data structure is not versioned within Snowplow Console. Changes to it will be published by this command.

```text
snowplow-cli data-structures publish dev [paths...] default: [./data-structures] [flags]
```

### Examples

```text
  $ snowplow-cli ds publish dev

  $ snowplow-cli ds publish dev --dry-run

  $ snowplow-cli ds publish dev --dry-run ./my-data-structures ./my-other-data-structures
```

### Options

```text
  -d, --dry-run       Only print planned changes without performing them

      --gh-annotate   Output suitable for github workflow annotation (ignores -s)

  -h, --help          help for dev
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Structures Publish Prod

Publish data structures to your production environment

### Synopsis

Publish data structures from your development to your production environment

Data structures found on \<path...> which are deployed to your development environment will be published to your production environment.

```text
snowplow-cli data-structures publish prod [paths...] default: [./data-structures] [flags]
```

### Examples

```text


	$ snowplow-cli ds publish prod

	$ snowplow-cli ds publish prod --dry-run

	$ snowplow-cli ds publish prod --dry-run ./my-data-structures ./my-other-data-structures

	
```

### Options

```text
  -d, --dry-run   Only print planned changes without performing them

  -h, --help      help for prod
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Data-Structures Validate

Validate data structures with Snowplow Console

### Synopsis

Sends all data structures from \<path> for validation by Snowplow Console.

```text
snowplow-cli data-structures validate [paths...] default: [./data-structures] [flags]
```

### Examples

```text
  $ snowplow-cli ds validate

  $ snowplow-cli ds validate ./my-data-structures ./my-other-data-structures
```

### Options

```text
      --gh-annotate   Output suitable for github workflow annotation (ignores -s)

  -h, --help          help for validate
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                              Then on:

                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                                Windows %AppData%\snowplow\snowplow.yml

      --debug                 Log output level to Debug

      --env-file string       Environment file (.env). Defaults to .env in current directory

                              Then on:

                                Unix $HOME/.config/snowplow/.env

                                Darwin $HOME/Library/Application Support/snowplow/.env

                                Windows %AppData%\snowplow\.env

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

      --json-output           Log output as json

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

  -q, --quiet                 Log output level to Warn

  -s, --silent                Disable output
```

## Mcp

Start an MCP (Model Context Protocol) stdio server for Snowplow validation and context

### Synopsis

Start an MCP (Model Context Protocol) stdio server that provides tools for:

- Validating Snowplow files (data-structures, data-products, source-applications)
- Retrieving the built-in schema and rules that define how Snowplow data structures, tracking plans, and source applications should be structured

```text
snowplow-cli mcp [flags]
```

### Examples

```text


  Claude Desktop config:

  {

    "mcpServers": {

      ...

      "snowplow-cli": {

        "command": "snowplow-cli", "args": ["mcp"]

      }

    }

  }



  VS Code '\<workspace\>/.vscode/mcp.json':

  {

    "servers": {

      ...

      "snowplow-cli": {

        "type": "stdio",

        "command": "snowplow-cli", "args": ["mcp"]

      }

    }

  }



  Cursor '\<workspace\>/.cursor/mcp.json':

  {

    "mcpServers": {

      ...

      "snowplow-cli": {

        "command": "snowplow-cli", "args": ["mcp", "--base-directory", "."]

      }

    }

  }



Note:

  This server's validation tools require filesystem paths to validate assets. For full

  functionality, your MCP client needs filesystem write access so created assets can be

  saved as files and then validated.



Setup options:

  - Enable filesystem access in your MCP client, or

  - Run alongside an MCP filesystem server (e.g., @modelcontextprotocol/server-filesystem)
```

### Options

```text
  -S, --api-key string          Snowplow Console api key

  -a, --api-key-id string       Snowplow Console api key id

      --base-directory string   The base path to use for relative file lookups. Useful for clients that pass in relative file paths.

      --dump-context            Dumps the result of the get_context tool to stdout and exits.

  -h, --help                    help for mcp

  -H, --host string             Snowplow Console host (default "https://console.snowplowanalytics.com")

  -m, --managed-from string     Link to a github repo where the data structure is managed

  -o, --org-id string           Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                          Then on:

                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                            Windows %AppData%\snowplow\snowplow.yml

      --debug             Log output level to Debug

      --env-file string   Environment file (.env). Defaults to .env in current directory

                          Then on:

                            Unix $HOME/.config/snowplow/.env

                            Darwin $HOME/Library/Application Support/snowplow/.env

                            Windows %AppData%\snowplow\.env

      --json-output       Log output as json

  -q, --quiet             Log output level to Warn

  -s, --silent            Disable output
```

## Setup

Set up Snowplow CLI with device authentication

### Synopsis

Authenticate with Snowplow Console using device authentication flow and create an API key

```text
snowplow-cli setup [flags]
```

### Examples

```text
  $ snowplow-cli setup

  $ snowplow-cli setup --read-only
```

### Options

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

      --auth0-domain string   Auth0 domain (default "id.snowplowanalytics.com")

      --client-id string      Auth0 Client ID for device auth (default "EXQ3csSDr6D7wTIiebNPhXpgkSsOzCzi")

      --dotenv                Store as .env file in current working directory

  -h, --help                  help for setup

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id

      --read-only             Create a read-only API key
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                          Then on:

                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                            Windows %AppData%\snowplow\snowplow.yml

      --debug             Log output level to Debug

      --env-file string   Environment file (.env). Defaults to .env in current directory

                          Then on:

                            Unix $HOME/.config/snowplow/.env

                            Darwin $HOME/Library/Application Support/snowplow/.env

                            Windows %AppData%\snowplow\.env

      --json-output       Log output as json

  -q, --quiet             Log output level to Warn

  -s, --silent            Disable output
```

## Status

Check Snowplow CLI configuration and connectivity

### Synopsis

Verify that the CLI is properly configured and can connect to Snowplow Console

```text
snowplow-cli status [flags]
```

### Examples

```text
  $ snowplow-cli status
```

### Options

```text
  -S, --api-key string        Snowplow Console api key

  -a, --api-key-id string     Snowplow Console api key id

  -h, --help                  help for status

  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")

  -m, --managed-from string   Link to a github repo where the data structure is managed

  -o, --org-id string         Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                          Then on:

                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                            Windows %AppData%\snowplow\snowplow.yml

      --debug             Log output level to Debug

      --env-file string   Environment file (.env). Defaults to .env in current directory

                          Then on:

                            Unix $HOME/.config/snowplow/.env

                            Darwin $HOME/Library/Application Support/snowplow/.env

                            Windows %AppData%\snowplow\.env

      --json-output       Log output as json

  -q, --quiet             Log output level to Warn

  -s, --silent            Disable output
```

## Events

Work with Snowplow events

### Examples

```text
  $ snowplow-cli events send --collector collector.example.com --sdjson '{"schema":"iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0","data":{"category":"test","action":"click"}}'
```

### Options

```text
  -h, --help   help for events
```

### Options inherited from parent commands

```text
      --config string   Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                        Then on:

                          Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                          Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                          Windows %AppData%\snowplow\snowplow.yml

      --debug           Log output level to Debug

      --json-output     Log output as json

  -q, --quiet           Log output level to Warn

  -s, --silent          Disable output
```

## Events Send

Send a single event to a Snowplow collector

### Synopsis

Send a single self-describing event to a Snowplow collector.

Provide either a full self-describing JSON via --sdjson, or a --schema URI plus a --json data payload. Optionally attach entities via --entities.

```text
snowplow-cli events send [flags]
```

### Examples

```text
  $ snowplow-cli events send -c collector.example.com -d iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0 -j '{"category":"test","action":"click"}'

  $ snowplow-cli events send -c collector.example.com -J '{"schema":"iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0","data":{"category":"test","action":"click"}}'
```

### Options

```text
  -a, --app-id string       Application ID (default "snowplowcli")

  -c, --collector string    Collector domain, e.g. collector.example.com (required)

  -e, --entities string     JSON array of self-describing JSON entities to attach (default "[]")

  -h, --help                help for send

  -i, --ip-address string   Custom IP address to track

  -j, --json string         Non-self-describing JSON data of the form {...}

  -m, --method string       HTTP method [POST|GET] (default "POST")

  -p, --protocol string     Protocol [http|https] (default "https")

  -d, --schema string       Schema (data structure) URI, of the form iglu:...

  -J, --sdjson string       Self-describing JSON of the form {"schema":"iglu:...","data":{...}}
```

### Options inherited from parent commands

```text
      --config string   Config file. Defaults to $HOME/.config/snowplow/snowplow.yml

                        Then on:

                          Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml

                          Darwin $HOME/Library/Application Support/snowplow/snowplow.yml

                          Windows %AppData%\snowplow\snowplow.yml

      --debug           Log output level to Debug

      --json-output     Log output as json

  -q, --quiet           Log output level to Warn

  -s, --silent          Disable output
```

---

# Send events via Snowplow CLI
> Send Snowplow events to a Collector from the command line with the Snowplow CLI events send command. Useful for sending a test event, verifying a pipeline, or adding tracking to shell scripts and terminal sessions without writing tracker code.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-cli/send-events/

The `events` subcommand of [Snowplow CLI](/docs/api-reference/snowplow-cli/) sends events to a Snowplow Collector from the command line. Use it to embed Snowplow tracking into your shell scripts and terminal sessions, or to send a quick test event while validating a pipeline.

Each event is sent as an individual payload, and the command exits once the Collector has responded.

## Prerequisites

You need [Snowplow CLI](/docs/api-reference/snowplow-cli/) installed. Sending events does not require Console credentials, so you can skip the configuration step if you only use the `events` subcommand.

## Send an event

The `events send` command sends a single [self-describing event](/docs/fundamentals/events/#self-describing-events) to your Collector. You can either pass a complete self-describing JSON, or pass a schema URI and its data separately and let the CLI assemble the event for you.

The only required flag is `--collector`, the domain of your Snowplow Collector. The examples below use the built-in `com.snowplowanalytics.snowplow/custom_event` schema, so they produce a valid event as long as you point `--collector` at a real Collector.

### Send a self-describing JSON

Pass a complete self-describing JSON with `--sdjson`:

```bash
snowplow-cli events send \

  --collector collector.example.com \

  --sdjson '{"schema":"iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0","data":{"category":"test","action":"click"}}'
```

### Send a schema and data separately

Pass the schema URI with `--schema` and a plain (non-self-describing) JSON payload with `--json`. The CLI combines them into a self-describing event:

```bash
snowplow-cli events send \

  --collector collector.example.com \

  --schema iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0 \

  --json '{"category":"test","action":"click"}'
```

### Attach entities

Attach [entities](/docs/fundamentals/entities/) with `--entities`, passing a JSON array of self-describing JSON objects:

```bash
snowplow-cli events send \

  --collector collector.example.com \

  --sdjson '{"schema":"iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0","data":{"category":"test","action":"click"}}' \

  --entities '[{"schema":"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0","data":{"id":"550e8400-e29b-41d4-a716-446655440000"}}]'
```

## Command options

Set the application ID with `--app-id`, the HTTP method with `--method` (defaults to `POST`), and the protocol with `--protocol` (defaults to `https`). For the full list of flags, see the [`events send` command reference](/docs/api-reference/snowplow-cli/reference/#events-send).

## Migrate from the Snowplow Tracking CLI

The standalone Snowplow Tracking CLI (`snowplow-tracking-cli`) has been merged into Snowplow CLI as the `events send` command. The old tool is deprecated and its repository will be archived. Migrate any scripts that use `snowplow-tracking-cli` to `snowplow-cli events send`.

The flags are largely the same, with a few renames and one changed default:

| Snowplow Tracking CLI | Snowplow CLI `events send` | Notes                                |
| --------------------- | -------------------------- | ------------------------------------ |
| `--collector`         | `--collector`              | Unchanged, required                  |
| `--appid`             | `--app-id`                 | Renamed; defaults to `snowplowcli`   |
| `--method`            | `--method`                 | Default changed from `GET` to `POST` |
| `--protocol`          | `--protocol`               | Unchanged, defaults to `https`       |
| `--sdjson`            | `--sdjson`                 | Unchanged                            |
| `--schema`            | `--schema`                 | Unchanged                            |
| `--json`              | `--json`                   | Unchanged                            |
| `--ipaddress`         | `--ip-address`             | Renamed                              |
| `--contexts`          | `--entities`               | Renamed                              |

For example, this Snowplow Tracking CLI command:

```bash
snowplow-tracking-cli --collector collector.example.com --appid myappid --method POST --sdjson '{"schema":"iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0","data":{"category":"test","action":"click"}}'
```

becomes:

```bash
snowplow-cli events send --collector collector.example.com --app-id myappid --sdjson '{"schema":"iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0","data":{"category":"test","action":"click"}}'
```

---

# Managing tracking plans via the CLI
> Use the Snowplow CLI data-products command to create, download, validate, sync, and release tracking plans, event specifications, and source applications with git-ops workflows.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-cli/tracking-plans/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

The `data-products` subcommand of [Snowplow CLI](/docs/api-reference/snowplow-cli/) provides a collection of functionality to ease the integration of custom development and publishing workflows.

## Snowplow CLI Prerequisites

Installed and configured [Snowplow CLI](/docs/api-reference/snowplow-cli/)

## Available commands

### Creating tracking plan

```bash
snowplow-cli dp generate --data-product my-data-product
```

This command creates a minimal tracking plan template in a new file `./data-products/my-data-product.yaml`.

### Creating source application

```bash
snowplow-cli dp generate --source-app my-source-app
```

This command creates a minimal source application template in a new file `./data-products/source-apps/my-source-app.yaml`.

### Creating event specification

To create an event specification, you need to modify the existing data-product file and add an event specification object. Here's a minimal example:

```yaml
apiVersion: v1

resourceType: data-product

resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0

data:

    name: test-cli

    sourceApplications: []

    eventSpecifications:

        - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca

          name: test

          event:

            source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0
```

> **Note:** The `source` fields of events and entities must refer to a deployed data structure. Referring to a locally created data structure is not yet supported.

### Linking tracking plan to a source application

To link a tracking plan to a source application, provide a list of references to the source application files in the `data.sourceApplications` field. Here's an example:

```yaml
apiVersion: v1

resourceType: data-product

resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0

data:

    name: test-cli

    sourceApplications:

        - $ref: ./source-apps/my-source-app.yaml
```

### Modifying the event specifications source applications

By default event specifications inherit all the source applications of the tracking plan. If you want to customise it, you can use the `excludedSourceApplications` in the event specification description to remove a given source application from an event specification.

```yaml
apiVersion: v1

resourceType: data-product

resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0

data:

    name: test-cli

    sourceApplications:

        - $ref: ./source-apps/generic.yaml

        - $ref: ./source-apps/specific.yaml

    eventSpecifications:

        - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca

          name: All source apps

          event:

            source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0

        - resourceName: b9c994a0-03b2-479c-b1cf-7d25c3adc572

          name: Not quite everything

          excludedSourceApplications:

            - $ref: ./source-apps/specific.yaml

          event:

            source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0
```

In this example event specification `All source apps` is related to both `generic` and `specific` source apps, but event specification `Not quite everything` is related only to the `generic` source application.

### Downloading tracking plans, event specifications and source apps

```bash
snowplow-cli dp download
```

This command retrieves all organization tracking plans, event specifications, and source applications. By default, it creates a folder named `data-products` in your current working directory. You can specify a different folder name as an argument if needed. The command creates the following structure:

- A main `data-products` folder containing your tracking plan files
- A `source-apps` subfolder containing source application definitions
- Event specifications embedded within their related tracking plan files.

### Validating tracking plans, event specifications and source applications

```bash
snowplow-cli dp validate
```

This command scans all files under `./data-products` and validates them using Snowplow Console. It checks:

1. Whether each file is in a valid format (YAML/JSON) with correctly formatted fields
2. Whether all source application references in the tracking plan files are valid
3. Whether event specification rules are compatible with their schemas If validation fails, the command displays the errors in the console and exits with status code 1.

### Syncing tracking plans, event specifications and source applications

```bash
snowplow-cli dp sync
```

This command locates all files under `./data-products`, validates them, and pushes local changes to Console. Tracking plans and source applications are updated in place. For event specifications, a new version in draft status may be created if the change is structural (name change, event change, or rules change).

> **Note:** The old `publish` command has been renamed to `sync`. Running `dp publish` still works as an alias for backward compatibility, but it may be removed in a future release. Update your scripts to use `dp sync` instead.

### Releasing event specifications

```bash
snowplow-cli dp release
```

This command first syncs local files with Console (like `sync`), then releases any draft event specifications. Releasing marks event specifications as published and enables event spec inference. Only event specifications that are part of the local tracking plan files are affected — other event specifications in Console are left unchanged. Event specifications without an event are skipped.

Use `sync` if you only want to push changes without changing event specification status.

---

# Snowplow Micro REST API
> Snowplow Micro REST API endpoints for querying events.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-micro/api/

This page documents the REST API of [Snowplow Micro](/docs/testing/snowplow-micro/).

## Authentication

When running Micro locally, no authentication is required. You can query all endpoints directly.

When using [Micro in Console](/docs/testing/snowplow-micro/console/), you need to authenticate with a Console access token. First, [create an API key](/docs/account-management/#create-an-api-key) and use it to [obtain an access token](/docs/account-management/#obtain-an-access-token). Then pass the token in the `Authorization` header:

```bash
curl -H 'Authorization: Bearer <JWT>' \

  https://<CONSOLE_MICRO_URL>/micro/events
```

## /micro/events

This endpoint queries events with server-side filtering, sorting, and pagination. It handles both valid and failed events.

### HTTP method

`POST` (`Content-Type: application/json`)

### Request format

| Field         | Type            | Required | Description                                                                                                                                        |
| ------------- | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `page`        | integer         | Yes      | Page number (1-indexed).                                                                                                                           |
| `pageSize`    | integer         | Yes      | Number of events per page. Capped at 100 when persistent storage is enabled (including [Micro in Console](/docs/testing/snowplow-micro/console/)). |
| `filters`     | array           | Yes      | List of column filters (see below).                                                                                                                |
| `validEvents` | boolean or null | No       | `true` for valid events only, `false` for failed events only, `null` (default) for all events.                                                     |
| `timeRange`   | object or null  | No       | Time range filter (see below).                                                                                                                     |
| `sorting`     | object or null  | No       | Sorting specification (see below).                                                                                                                 |

Each object in the `filters` array has the following fields:

| Field                          | Type             | Description                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `column`                       | string           | Column name to filter on (e.g., `app_id`, `event_name`).                                                                                                                                                                                                                                                                                                                           |
| `values` (since version 4.4.0) | array of strings | One or more values to filter for. In-memory mode uses case-insensitive substring matching. When persistent storage is enabled (including [Micro in Console](/docs/testing/snowplow-micro/console/)), exact matching is used and only the following columns are supported: `event_id`, `app_id`, `event_name`, `platform`, `name_tracker`, `user_id`, `domain_userid`, `v_tracker`. |
| `value` (before version 4.4.0) | string           | The value to filter for. In-memory mode uses case-insensitive substring matching. When persistent storage is enabled (including [Micro in Console](/docs/testing/snowplow-micro/console/)), exact matching is used and only the following columns are supported: `event_id`, `app_id`, `event_name`, `platform`, `name_tracker`, `user_id`, `domain_userid`, `v_tracker`.          |

The `timeRange` object has the following fields:

| Field   | Type                      | Description                 |
| ------- | ------------------------- | --------------------------- |
| `start` | string (ISO 8601) or null | Start timestamp, inclusive. |
| `end`   | string (ISO 8601) or null | End timestamp, exclusive.   |

The `sorting` object has the following fields:

| Field    | Type    | Description                                                                                                                                                                                                                                                                                         |
| -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `column` | string  | Column to sort by. When persistent storage is enabled (including [Micro in Console](/docs/testing/snowplow-micro/console/)), only the following columns are supported: `event_id`, `app_id`, `event_name`, `platform`, `name_tracker`, `user_id`, `domain_userid`, `v_tracker`, `collector_tstamp`. |
| `desc`   | boolean | `true` for descending, `false` for ascending.                                                                                                                                                                                                                                                       |

Example request:

```bash
curl -X POST http://localhost:9090/micro/events \

  -H 'Content-Type: application/json' \

  -d '{

    "filters": [

      {"column": "app_id", "value": "myapp"}

    ],

    "validEvents": true,

    "timeRange": {

      "start": "2025-01-01T00:00:00Z",

      "end": "2025-01-02T00:00:00Z"

    },

    "sorting": {"column": "collector_tstamp", "desc": true},

    "page": 1,

    "pageSize": 50

  }'
```

### Response format

| Field        | Type    | Description                                                                                                          |
| ------------ | ------- | -------------------------------------------------------------------------------------------------------------------- |
| `events`     | array   | Array of event objects in the [Analytics SDK](https://github.com/snowplow/snowplow-scala-analytics-sdk) JSON format. |
| `totalPages` | integer | Total number of pages available.                                                                                     |
| `totalItems` | integer | Total number of events matching the filters.                                                                         |

Example response with a valid event:

```json
{

  "events": [

    {

      "event_id": "bee0a6d7-fc17-4392-b2bc-2208e8e944f3",

      "event_name": "page_view",

      "app_id": "myapp",

      "collector_tstamp": "2025-01-01T12:30:45.123Z",

      "platform": "web",

      "v_tracker": "js-4.0.0"

    }

  ],

  "totalPages": 5,

  "totalItems": 234

}
```

Failed events (queried with `"validEvents": false`) have the same JSON structure, but include a `contexts_com_snowplowanalytics_snowplow_failure_1` field with error details. For example, an event that failed due to an unresolved schema:

```json
{

  "events": [

    {

      "event_id": "1ebca17a-0e59-4848-ba3a-50ff6ac8ace7",

      "event": "unstruct",

      "app_id": "myapp",

      "collector_tstamp": "2026-05-22T10:45:21.287Z",

      "platform": "web",

      "v_tracker": "js-4.0.0",

      "contexts_com_snowplowanalytics_snowplow_failure_1": [

        {

          "data": {

            "foo": "bar"

          },

          "errors": [

            {

              "source": "unstruct",

              "message": "Resolution error: schema iglu:com.example/missing/jsonschema/1-0-0 not found",

              "lookupHistory": [

                {

                  "errors": [{"error": "NotFound"}],

                  "attempts": 1,

                  "repository": "Iglu Central",

                  "lastAttempt": "2026-05-22T10:45:22.281Z"

                },

                {

                  "errors": [{"error": "NotFound"}],

                  "attempts": 1,

                  "repository": "Iglu Client Embedded",

                  "lastAttempt": "2026-05-22T10:45:22.216Z"

                }

              ]

            }

          ],

          "schema": "iglu:com.example/missing/jsonschema/1-0-0",

          "timestamp": "2026-05-22T10:45:22.215Z",

          "failureType": "ResolutionError",

          "componentName": "snowplow-micro",

          "componentVersion": "4.2.0",

          "_schema_version": "iglu:com.snowplowanalytics.snowplow/failure/jsonschema/1-0-0"

        }

      ]

    }

  ],

  "totalPages": 1,

  "totalItems": 1

}
```

## /micro/timeline

This endpoint returns event counts aggregated into time buckets. The Micro UI uses it to display event timelines.

### HTTP method

`POST` (`Content-Type: application/json`)

### Request format

The request body contains an array of time buckets to aggregate events into.

| Field     | Type  | Required | Description           |
| --------- | ----- | -------- | --------------------- |
| `buckets` | array | Yes      | List of time buckets. |

Each object in the `buckets` array has the following fields:

| Field   | Type              | Description                     |
| ------- | ----------------- | ------------------------------- |
| `start` | string (ISO 8601) | Start of the bucket, inclusive. |
| `end`   | string (ISO 8601) | End of the bucket, exclusive.   |

Example request:

```bash
curl -X POST http://localhost:9090/micro/timeline \

  -H 'Content-Type: application/json' \

  -d '{

    "buckets": [

      {"start": "2025-01-01T00:00:00Z", "end": "2025-01-01T01:00:00Z"},

      {"start": "2025-01-01T01:00:00Z", "end": "2025-01-01T02:00:00Z"}

    ]

  }'
```

### Response format

| Field    | Type  | Description                                         |
| -------- | ----- | --------------------------------------------------- |
| `points` | array | One data point per input bucket, in the same order. |

Each object in the `points` array has the following fields:

| Field          | Type    | Description                                         |
| -------------- | ------- | --------------------------------------------------- |
| `validEvents`  | integer | Count of valid events in the bucket.                |
| `failedEvents` | integer | Count of failed events in the bucket.               |
| `bucket`       | object  | The original bucket definition (`start` and `end`). |

Example:

```json
{

  "points": [

    {

      "validEvents": 42,

      "failedEvents": 3,

      "bucket": {

        "start": "2025-01-01T00:00:00Z",

        "end": "2025-01-01T01:00:00Z"

      }

    },

    {

      "validEvents": 58,

      "failedEvents": 1,

      "bucket": {

        "start": "2025-01-01T01:00:00Z",

        "end": "2025-01-01T02:00:00Z"

      }

    }

  ]

}
```

## /micro/all

> **Note:** This endpoint is not available when using Micro [through Snowplow Console](/docs/testing/snowplow-micro/console/) (or when using persistent storage in general).

This endpoint responds with a summary JSON object of the number of total, good and bad events currently in the cache.

### HTTP method

`GET`, `POST`, `OPTIONS`

### Response format

Example:

```json
{

  "total": 7,

  "good": 5,

  "bad": 2

}
```

## /micro/good

> **Note:** This endpoint is not available when using Micro [through Snowplow Console](/docs/testing/snowplow-micro/console/) (or when using persistent storage in general).

This endpoint queries the good events, which are the events that have been successfully validated.

### HTTP method

- `GET`: get _all_ the good events from the cache.
- `POST`: get the good events with the possibility to filter.

### Response format

JSON array of [GoodEvent](https://github.com/snowplow/snowplow-micro/blob/master/src/main/scala/com.snowplowanalytics.snowplow.micro/model.scala#L19)s. A `GoodEvent` contains 4 fields:

- `rawEvent`: contains the [RawEvent](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/adapters/RawEvent.scala#L28). It corresponds to the format of a validated event just before being enriched.
- `event`: contains the [canonical snowplow Event](https://github.com/snowplow/snowplow-scala-analytics-sdk/blob/master/src/main/scala/com.snowplowanalytics.snowplow.analytics.scalasdk/Event.scala#L42). It is in the format of an event after enrichment, even if all the enrichments are deactivated.
- `eventType`: type of the event.
- `schema`: schema of the event in case of a self-describing event.
- `contexts`: entities attached to the event.

An example of a response with one event can be found below:

```json
[

  {

    "rawEvent":{

      "api":{

        "vendor":"com.snowplowanalytics.snowplow",

        "version":"tp2"

      },

      "parameters":{

        "e":"pv",

        "duid":"36746fd2-8441-4ea2-8ad0-237d6f4c77cf",

        "vid":"1",

        "co":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"5cea9899-10df-4ccf-bf66-8c36f4a4bba2\"}}]}",

        "eid":"bee0a6d7-fc17-4392-b2bc-2208e8e944f3",

        "url":"http://localhost:8000/",

        "refr":"http://localhost:8000/__/",

        "aid":"shop",

        "tna":"sp1",

        "cs":"UTF-8",

        "cd":"24",

        "stm":"1630238465752",

        "tz":"Europe/London",

        "tv":"js-3.1.3",

        "vp":"1000x660",

        "ds":"988x670",

        "res":"1920x1080",

        "cookie":"1",

        "p":"web",

        "dtm":"1630238465748",

        "uid":"tester",

        "lang":"en-US",

        "sid":"6d15a4fb-9623-4ba1-b876-5240e72e6970"

      },

      "contentType":"application/json",

      "source":{

        "name":"ssc-2.3.1-stdout$",

        "encoding":"UTF-8",

        "hostname":"0.0.0.0"

      },

      "context":{

        "timestamp":"2021-08-29T12:01:05.787Z",

        "ipAddress":"172.17.0.1",

        "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

        "refererUri":"http://localhost:8000/",

        "headers":[

          "Timeout-Access: <function1>",

          "Connection: keep-alive",

          "Host: 0.0.0.0:9090",

          "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

          "Accept: */*",

          "Accept-Language: en-US, en;q=0.5",

          "Accept-Encoding: gzip",

          "Referer: http://localhost:8000/",

          "Origin: http://localhost:8000",

          "Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6",

          "application/json"

        ],

        "userId":"3734601f-5c3d-47c5-b367-0883e1ed74e6"

      }

    },

    "eventType":"page_view",

    "schema":"iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0",

    "contexts":[

      "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

    ],

    "event":{

      "app_id":"shop",

      "platform":"web",

      "etl_tstamp":"2021-08-29T12:01:05.792Z",

      "collector_tstamp":"2021-08-29T12:01:05.787Z",

      "dvce_created_tstamp":"2021-08-29T12:01:05.748Z",

      "event":"page_view",

      "event_id":"bee0a6d7-fc17-4392-b2bc-2208e8e944f3",

      "txn_id":null,

      "name_tracker":"sp1",

      "v_tracker":"js-3.1.3",

      "v_collector":"ssc-2.3.1-stdout$",

      "v_etl":"snowplow-micro-1.3.1-common-2.0.2",

      "user_id":"tester",

      "user_ipaddress":"172.17.0.1",

      "user_fingerprint":null,

      "domain_userid":"36746fd2-8441-4ea2-8ad0-237d6f4c77cf",

      "domain_sessionidx":1,

      "network_userid":"3734601f-5c3d-47c5-b367-0883e1ed74e6",

      "geo_country":null,

      "geo_region":null,

      "geo_city":null,

      "geo_zipcode":null,

      "geo_latitude":null,

      "geo_longitude":null,

      "geo_region_name":null,

      "ip_isp":null,

      "ip_organization":null,

      "ip_domain":null,

      "ip_netspeed":null,

      "page_url":"http://localhost:8000/",

      "page_title":null,

      "page_referrer":"http://localhost:8000/__/",

      "page_urlscheme":"http",

      "page_urlhost":"localhost",

      "page_urlport":8000,

      "page_urlpath":"/",

      "page_urlquery":null,

      "page_urlfragment":null,

      "refr_urlscheme":"http",

      "refr_urlhost":"localhost",

      "refr_urlport":8000,

      "refr_urlpath":"/__/",

      "refr_urlquery":null,

      "refr_urlfragment":null,

      "refr_medium":null,

      "refr_source":null,

      "refr_term":null,

      "mkt_medium":null,

      "mkt_source":null,

      "mkt_term":null,

      "mkt_content":null,

      "mkt_campaign":null,

      "contexts":{

        "schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",

        "data":[

          {

            "schema":"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",

            "data":{

              "id":"5cea9899-10df-4ccf-bf66-8c36f4a4bba2"

            }

          }

        ]

      },

      "se_category":null,

      "se_action":null,

      "se_label":null,

      "se_property":null,

      "se_value":null,

      "unstruct_event":null,

      "tr_orderid":null,

      "tr_affiliation":null,

      "tr_total":null,

      "tr_tax":null,

      "tr_shipping":null,

      "tr_city":null,

      "tr_state":null,

      "tr_country":null,

      "ti_orderid":null,

      "ti_sku":null,

      "ti_name":null,

      "ti_category":null,

      "ti_price":null,

      "ti_quantity":null,

      "pp_xoffset_min":null,

      "pp_xoffset_max":null,

      "pp_yoffset_min":null,

      "pp_yoffset_max":null,

      "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

      "br_name":null,

      "br_family":null,

      "br_version":null,

      "br_type":null,

      "br_renderengine":null,

      "br_lang":"en-US",

      "br_features_pdf":null,

      "br_features_flash":null,

      "br_features_java":null,

      "br_features_director":null,

      "br_features_quicktime":null,

      "br_features_realplayer":null,

      "br_features_windowsmedia":null,

      "br_features_gears":null,

      "br_features_silverlight":null,

      "br_cookies":true,

      "br_colordepth":"24",

      "br_viewwidth":1000,

      "br_viewheight":660,

      "os_name":null,

      "os_family":null,

      "os_manufacturer":null,

      "os_timezone":"Europe/London",

      "dvce_type":null,

      "dvce_ismobile":null,

      "dvce_screenwidth":1920,

      "dvce_screenheight":1080,

      "doc_charset":"UTF-8",

      "doc_width":988,

      "doc_height":670,

      "tr_currency":null,

      "tr_total_base":null,

      "tr_tax_base":null,

      "tr_shipping_base":null,

      "ti_currency":null,

      "ti_price_base":null,

      "base_currency":null,

      "geo_timezone":null,

      "mkt_clickid":null,

      "mkt_network":null,

      "etl_tags":null,

      "dvce_sent_tstamp":"2021-08-29T12:01:05.752Z",

      "refr_domain_userid":null,

      "refr_dvce_tstamp":null,

      "derived_contexts":{},

      "domain_sessionid":"6d15a4fb-9623-4ba1-b876-5240e72e6970",

      "derived_tstamp":"2021-08-29T12:01:05.783Z",

      "event_vendor":"com.snowplowanalytics.snowplow",

      "event_name":"page_view",

      "event_format":"jsonschema",

      "event_version":"1-0-0",

      "event_fingerprint":null,

      "true_tstamp":null

    }

  }

]
```

### Filters

When querying `/micro/good` with `POST` (`Content-Type: application/json` needs to be set in the headers of the request), it's possible to specify filters, thanks to a JSON in the data of the HTTP request.

Example of command to query the good events: 

```bash
curl -X POST -H 'Content-Type: application/json' <IP:PORT>/micro/good -d '<JSON>'
```

An example of JSON with filters could be:

```json
{

  "schema": "iglu:com.acme/example/jsonschema/1-0-0",

  "contexts": [

    "com.snowplowanalytics.mobile/application/jsonschema/1-0-0",

    "com.snowplowanalytics.mobile/screen/jsonschema/1-0-0"

  ],

  "limit": 10

}
```

List of possible fields for the filters:

- `event_type`: type of the event (in `e` param);
- `schema`: corresponds to the schema of a [self-describing event](/docs/fundamentals/events/#self-describing-events) (schema of the self-describing JSON contained in `ue_pr` or `ue_px`). It automatically implies `event_type` = `ue`.
- `contexts`: list of the schemas contained in the entities of an event (parameters `co` or `cx`). An event must contain **all** the entities in the list to be returned. It can also contain more entities than the ones specified in the request.
- `limit`: limit the number of events in the response (most recent events are returned).

It's not necessary to specify all the fields in a request, only the ones that need to be used for filtering.

## /micro/bad

> **Note:** This endpoint is not available when using Micro [through Snowplow Console](/docs/testing/snowplow-micro/console/) (or when using persistent storage in general).

This endpoint queries the bad events, which are the events that failed validation.

### HTTP method

- `GET`: get _all_ the bad events from the cache.
- `POST`: get the bad events with the possibility to filter.

### Response format

JSON array of [BadEvent](https://github.com/snowplow/snowplow-micro/blob/master/src/main/scala/com.snowplowanalytics.snowplow.micro/model.scala#L28)s. A `BadEvent` contains 3 fields:

- `collectorPayload`: contains the [CollectorPayload](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/loaders/CollectorPayload.scala#L107) with all the raw information of the tracking event. This field can be empty if an error occured before trying to validate a payload.
- `rawEvent`: contains the [RawEvent](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/adapters/RawEvent.scala#L28). It corresponds to the format of a validated event just before being enriched.
- `errors`: list of errors that occured during the validation of the tracking event.

An example of a response with one bad event can be found below:

```json
[

  {

    "collectorPayload":{

      "api":{

        "vendor":"com.snowplowanalytics.snowplow",

        "version":"tp2"

      },

      "querystring":[],

      "contentType":"application/json",

      "body":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4\",\"data\":[{\"e\":\"ue\",\"eid\":\"36c39024-7b1b-4c2c-ae85-e95a8cb8340a\",\"tv\":\"js-3.1.3\",\"tna\":\"spmicro\",\"aid\":\"sh0pspr33\",\"p\":\"web\",\"cookie\":\"1\",\"cs\":\"UTF-8\",\"lang\":\"en-US\",\"res\":\"1920x1080\",\"cd\":\"24\",\"tz\":\"Europe/London\",\"dtm\":\"1630234190717\",\"vp\":\"1000x660\",\"ds\":\"1003x2242\",\"vid\":\"1\",\"sid\":\"13c8f5ac-d999-4923-940d-b39f7b74aa94\",\"duid\":\"8a17bb29-e35c-4363-aec3-85b9b363f9bf\",\"uid\":\"tester\",\"refr\":\"http://localhost:8000/\",\"url\":\"http://localhost:8000/shop/\",\"ue_pr\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"schema\\\":\\\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"type\\\":\\\"add\\\"}}}\",\"co\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\\\",\\\"data\\\":[{\\\"schema\\\":\\\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"sku\\\":\\\"hh456\\\",\\\"name\\\":\\\"One-size bucket hat\\\",\\\"price\\\":24.49,\\\"quantity\\\":\\\"2\\\"}},{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"id\\\":\\\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\\\"}}]}\",\"stm\":\"1630234190719\"}]}",

      "source":{

        "name":"ssc-2.3.1-stdout$",

        "encoding":"UTF-8",

        "hostname":"0.0.0.0"

      },

      "context":{

        "timestamp":"2021-08-29T10:49:50.727Z",

        "ipAddress":"172.17.0.1",

        "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

        "refererUri":"http://localhost:8000/",

        "headers":[

          "Timeout-Access: <function1>",

          "Connection: keep-alive",

          "Host: 0.0.0.0:9090",

          "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

          "Accept: */*",

          "Accept-Language: en-US, en;q=0.5",

          "Accept-Encoding: gzip",

          "Referer: http://localhost:8000/",

          "Origin: http://localhost:8000",

          "Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6",

          "application/json"

        ],

        "userId":"3734601f-5c3d-47c5-b367-0883e1ed74e6"

      }

    },

    "rawEvent":{

      "api":{

        "vendor":"com.snowplowanalytics.snowplow",

        "version":"tp2"

      },

      "parameters":{

        "e":"ue",

        "duid":"8a17bb29-e35c-4363-aec3-85b9b363f9bf",

        "vid":"1",

        "co":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\",\"data\":{\"sku\":\"hh456\",\"name\":\"One-size bucket hat\",\"price\":24.49,\"quantity\":\"2\"}},{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\"}}]}",

        "eid":"36c39024-7b1b-4c2c-ae85-e95a8cb8340a",

        "url":"http://localhost:8000/shop/",

        "refr":"http://localhost:8000/",

        "aid":"sh0pspr33",

        "tna":"spmicro",

        "cs":"UTF-8",

        "cd":"24",

        "stm":"1630234190719",

        "tz":"Europe/London",

        "tv":"js-3.1.3",

        "vp":"1000x660",

        "ds":"1003x2242",

        "ue_pr":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\",\"data\":{\"schema\":\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\",\"data\":{\"type\":\"add\"}}}",

        "res":"1920x1080",

        "cookie":"1",

        "p":"web",

        "dtm":"1630234190717",

        "uid":"tester",

        "lang":"en-US",

        "sid":"13c8f5ac-d999-4923-940d-b39f7b74aa94"

      },

      "contentType":"application/json",

      "source":{

        "name":"ssc-2.3.1-stdout$",

        "encoding":"UTF-8",

        "hostname":"0.0.0.0"

      },

      "context":{

        "timestamp":"2021-08-29T10:49:50.727Z",

        "ipAddress":"172.17.0.1",

        "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

        "refererUri":"http://localhost:8000/",

        "headers":[

          "Timeout-Access: <function1>",

          "Connection: keep-alive",

          "Host: 0.0.0.0:9090",

          "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",

          "Accept: */*",

          "Accept-Language: en-US, en;q=0.5",

          "Accept-Encoding: gzip",

          "Referer: http://localhost:8000/",

          "Origin: http://localhost:8000",

          "Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6",

          "application/json"

        ],

        "userId":"3734601f-5c3d-47c5-b367-0883e1ed74e6"

      }

    },

    "errors":[

      "Error while validating the event",

      "{\"schema\":\"iglu:com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema/2-0-0\",\"data\":{\"processor\":{\"artifact\":\"snowplow-micro\",\"version\":\"1.2.1\"},\"failure\":{\"timestamp\":\"2021-08-29T10:49:50.739178Z\",\"messages\":[{\"schemaKey\":\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\",\"error\":{\"error\":\"ValidationError\",\"dataReports\":[{\"message\":\"$.quantity: string found, integer expected\",\"path\":\"$.quantity\",\"keyword\":\"type\",\"targets\":[\"string\",\"integer\"]}]}}]},\"payload\":{\"enriched\":{\"app_id\":\"sh0pspr33\",\"platform\":\"web\",\"etl_tstamp\":\"2021-08-29 10:49:50.731\",\"collector_tstamp\":\"2021-08-29 10:49:50.727\",\"dvce_created_tstamp\":\"2021-08-29 10:49:50.717\",\"event\":\"unstruct\",\"event_id\":\"36c39024-7b1b-4c2c-ae85-e95a8cb8340a\",\"txn_id\":null,\"name_tracker\":\"spmicro\",\"v_tracker\":\"js-3.1.3\",\"v_collector\":\"ssc-2.3.1-stdout$\",\"v_etl\":\"snowplow-micro-1.2.1-common-2.0.2\",\"user_id\":\"tester\",\"user_ipaddress\":\"172.17.0.1\",\"user_fingerprint\":null,\"domain_userid\":\"8a17bb29-e35c-4363-aec3-85b9b363f9bf\",\"domain_sessionidx\":1,\"network_userid\":\"3734601f-5c3d-47c5-b367-0883e1ed74e6\",\"geo_country\":null,\"geo_region\":null,\"geo_city\":null,\"geo_zipcode\":null,\"geo_latitude\":null,\"geo_longitude\":null,\"geo_region_name\":null,\"ip_isp\":null,\"ip_organization\":null,\"ip_domain\":null,\"ip_netspeed\":null,\"page_url\":\"http://localhost:8000/shop/\",\"page_title\":null,\"page_referrer\":\"http://localhost:8000/\",\"page_urlscheme\":null,\"page_urlhost\":null,\"page_urlport\":null,\"page_urlpath\":null,\"page_urlquery\":null,\"page_urlfragment\":null,\"refr_urlscheme\":null,\"refr_urlhost\":null,\"refr_urlport\":null,\"refr_urlpath\":null,\"refr_urlquery\":null,\"refr_urlfragment\":null,\"refr_medium\":null,\"refr_source\":null,\"refr_term\":null,\"mkt_medium\":null,\"mkt_source\":null,\"mkt_term\":null,\"mkt_content\":null,\"mkt_campaign\":null,\"contexts\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\\\",\\\"data\\\":[{\\\"schema\\\":\\\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"sku\\\":\\\"hh456\\\",\\\"name\\\":\\\"One-size bucket hat\\\",\\\"price\\\":24.49,\\\"quantity\\\":\\\"2\\\"}},{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"id\\\":\\\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\\\"}}]}\",\"se_category\":null,\"se_action\":null,\"se_label\":null,\"se_property\":null,\"se_value\":null,\"unstruct_event\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"schema\\\":\\\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"type\\\":\\\"add\\\"}}}\",\"tr_orderid\":null,\"tr_affiliation\":null,\"tr_total\":null,\"tr_tax\":null,\"tr_shipping\":null,\"tr_city\":null,\"tr_state\":null,\"tr_country\":null,\"ti_orderid\":null,\"ti_sku\":null,\"ti_name\":null,\"ti_category\":null,\"ti_price\":null,\"ti_quantity\":null,\"pp_xoffset_min\":null,\"pp_xoffset_max\":null,\"pp_yoffset_min\":null,\"pp_yoffset_max\":null,\"useragent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0\",\"br_name\":null,\"br_family\":null,\"br_version\":null,\"br_type\":null,\"br_renderengine\":null,\"br_lang\":\"en-US\",\"br_features_pdf\":null,\"br_features_flash\":null,\"br_features_java\":null,\"br_features_director\":null,\"br_features_quicktime\":null,\"br_features_realplayer\":null,\"br_features_windowsmedia\":null,\"br_features_gears\":null,\"br_features_silverlight\":null,\"br_cookies\":1,\"br_colordepth\":\"24\",\"br_viewwidth\":1000,\"br_viewheight\":660,\"os_name\":null,\"os_family\":null,\"os_manufacturer\":null,\"os_timezone\":\"Europe/London\",\"dvce_type\":null,\"dvce_ismobile\":null,\"dvce_screenwidth\":1920,\"dvce_screenheight\":1080,\"doc_charset\":\"UTF-8\",\"doc_width\":1003,\"doc_height\":2242,\"tr_currency\":null,\"tr_total_base\":null,\"tr_tax_base\":null,\"tr_shipping_base\":null,\"ti_currency\":null,\"ti_price_base\":null,\"base_currency\":null,\"geo_timezone\":null,\"mkt_clickid\":null,\"mkt_network\":null,\"etl_tags\":null,\"dvce_sent_tstamp\":\"2021-08-29 10:49:50.719\",\"refr_domain_userid\":null,\"refr_dvce_tstamp\":null,\"derived_contexts\":null,\"domain_sessionid\":\"13c8f5ac-d999-4923-940d-b39f7b74aa94\",\"derived_tstamp\":null,\"event_vendor\":null,\"event_name\":null,\"event_format\":null,\"event_version\":null,\"event_fingerprint\":null,\"true_tstamp\":null},\"raw\":{\"vendor\":\"com.snowplowanalytics.snowplow\",\"version\":\"tp2\",\"parameters\":[{\"name\":\"e\",\"value\":\"ue\"},{\"name\":\"duid\",\"value\":\"8a17bb29-e35c-4363-aec3-85b9b363f9bf\"},{\"name\":\"vid\",\"value\":\"1\"},{\"name\":\"co\",\"value\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\\\",\\\"data\\\":[{\\\"schema\\\":\\\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"sku\\\":\\\"hh456\\\",\\\"name\\\":\\\"One-size bucket hat\\\",\\\"price\\\":24.49,\\\"quantity\\\":\\\"2\\\"}},{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"id\\\":\\\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\\\"}}]}\"},{\"name\":\"eid\",\"value\":\"36c39024-7b1b-4c2c-ae85-e95a8cb8340a\"},{\"name\":\"url\",\"value\":\"http://localhost:8000/shop/\"},{\"name\":\"refr\",\"value\":\"http://localhost:8000/\"},{\"name\":\"aid\",\"value\":\"sh0pspr33\"},{\"name\":\"tna\",\"value\":\"spmicro\"},{\"name\":\"cs\",\"value\":\"UTF-8\"},{\"name\":\"cd\",\"value\":\"24\"},{\"name\":\"stm\",\"value\":\"1630234190719\"},{\"name\":\"tz\",\"value\":\"Europe/London\"},{\"name\":\"tv\",\"value\":\"js-3.1.3\"},{\"name\":\"vp\",\"value\":\"1000x660\"},{\"name\":\"ds\",\"value\":\"1003x2242\"},{\"name\":\"ue_pr\",\"value\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"schema\\\":\\\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"type\\\":\\\"add\\\"}}}\"},{\"name\":\"res\",\"value\":\"1920x1080\"},{\"name\":\"cookie\",\"value\":\"1\"},{\"name\":\"p\",\"value\":\"web\"},{\"name\":\"dtm\",\"value\":\"1630234190717\"},{\"name\":\"uid\",\"value\":\"tester\"},{\"name\":\"lang\",\"value\":\"en-US\"},{\"name\":\"sid\",\"value\":\"13c8f5ac-d999-4923-940d-b39f7b74aa94\"}],\"contentType\":\"application/json\",\"loaderName\":\"ssc-2.3.1-stdout$\",\"encoding\":\"UTF-8\",\"hostname\":\"0.0.0.0\",\"timestamp\":\"2021-08-29T10:49:50.727Z\",\"ipAddress\":\"172.17.0.1\",\"useragent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0\",\"refererUri\":\"http://localhost:8000/\",\"headers\":[\"Timeout-Access: <function1>\",\"Connection: keep-alive\",\"Host: 0.0.0.0:9090\",\"User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0\",\"Accept: */*\",\"Accept-Language: en-US, en;q=0.5\",\"Accept-Encoding: gzip\",\"Referer: http://localhost:8000/\",\"Origin: http://localhost:8000\",\"Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6\",\"application/json\"],\"userId\":\"3734601f-5c3d-47c5-b367-0883e1ed74e6\"}}}}"

    ]

  }

]
```

### Filters

When querying `/micro/bad` with `POST` (`Content-Type: application/json` needs to be set in the headers of the request), it's possible to specify filters, thanks to a JSON in the data of the HTTP request.

Example of command to query the bad events: 

```bash
curl -X POST -H 'Content-Type: application/json' <IP:PORT>/micro/bad -d '<JSON>'
```

An example of JSON with filters could be:

```json
{

    "vendor":"com.snowplowanalytics.snowplow",

    "version":"tp2",

    "limit": 10

}
```

List of possible fields for the filters:

- `vendor`: vendor for the tracking event.
- `version`: version of the vendor for the tracking event.
- `limit`: limit the number of events in the response (most recent events are returned).

It's not necessary to specify all the fields in each request, only the ones that need to be used for filtering.

## /micro/reset

Sending a request to this endpoint deletes all events stored by Micro.

### HTTP method

`GET`, `POST`

### Response format

Expected:

```json
{

  "total": 0,

  "good": 0,

  "bad": 0

}
```

## /micro/iglu

> **Note:** This is available since version 1.2.0.

The `/micro/iglu` endpoint can be used in order to check whether a schema can be resolved.

Schema lookup should be in format:

```text
/micro/iglu/{vendor}/{schemaName}/jsonschema/{schemaVersion}
```

Or more specifically:

```text
/micro/iglu/{vendor}/{schemaName}/jsonschema/{model}-{revision}-{addition}
```

For example, assuming Micro running on localhost port `9090`:

```bash
curl -X GET http://localhost:9090/micro/iglu/com.myvendor/myschema/jsonschema/1-0-0
```

### HTTP method

`GET`

### Response format

The JSON schema itself, if resolved:

```json
{

  "$schema":"http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

  "description":"A template for a self-describing JSON Schema for use with Iglu",

  "self": {

    "vendor":"com.myvendor",

    "name":"myschema",

    "format":"jsonschema",

    "version":"1-0-0"

  },

  "type":"object",

  "properties": {

    "myStringProperty": {

      "type":"string"

    },

    "myNumberProperty":{

      "type":"number"

    }

  },

  "required": ["myStringProperty","myNumberProperty"],

  "additionalProperties":false

}
```

If a schema cannot be resolved, a JSON indicating the Iglu repositories searched:

```json
{

  "value": {

    "Iglu Central": {

      "errors": [{"error":"NotFound"}],

      "attempts":1,

      "lastAttempt":"2021-08-26T13:41:06.905Z"

    },

    "Iglu Client Embedded": {

      "errors": [{"error":"NotFound"}],

      "attempts":1,

      "lastAttempt":"2021-08-26T13:41:06.677Z"

    }

  }

}
```

---

# Snowplow Micro API reference
> Snowplow Micro arguments and environment variables.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-micro/

See [this guide](/docs/testing/snowplow-micro/) for learning about Snowplow Micro and getting started with it.

> **Note:** You can skip this reference page if you are running Micro [through Console](/docs/testing/snowplow-micro/console/).

You can always run Micro with the `--help` argument to find out what is supported:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.4.0 --help
```

## Arguments

| Argument                              | Description                                                                                                                                                                                   |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--config`                            | Configuration file for Collector and Enrich ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-collector-and-enrich-configuration))                                    |
| `--iglu`                              | Configuration file for Iglu Client ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-iglu-resolver-configuration))                                                    |
| `-t`, `--output-tsv` _(since 1.4.0)_  | Print events in TSV format to standard output ([usage](/docs/testing/snowplow-micro/local/#exporting-events))                                                                                 |
| `-j`, `--output-json` _(since 2.4.0)_ | Print events in JSON format to standard output ([usage](/docs/testing/snowplow-micro/local/#exporting-events))                                                                                |
| `-d`, `--destination` _(since 2.4.0)_ | Send data to an HTTP endpoint instead of outputting it via standard output. Requires either `--output-tsv` or `--output-json` ([usage](/docs/testing/snowplow-micro/local/#exporting-events)) |
| `--yauaa`                             | Enable YAUAA user agent enrichment ([usage](/docs/testing/snowplow-micro/local/enrichments/#yauaa-yet-another-user-agent-analyzer))                                                           |
| `--no-storage` _(since 4.0.0)_        | Do not store the events anywhere and disable the API                                                                                                                                          |
| `--storage` _(since 4.0.0)_           | Enable PostgreSQL storage backend ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#persisting-events-across-restarts))                                                             |

## Environment variables

| Variable                     | Version | Description                                                                                                                                                                    |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MICRO_IGLU_REGISTRY_URL`    | 1.5.0+  | The URL for an additional custom Iglu registry ([usage](/docs/testing/snowplow-micro/local/schemas/#pointing-micro-to-an-iglu-registry))                                       |
| `MICRO_IGLU_API_KEY`         | 1.5.0+  | An optional API key for an Iglu registry defined with `MICRO_IGLU_REGISTRY_URL`                                                                                                |
| `MICRO_SSL_CERT_PASSWORD`    | 1.7.0+  | The password for the optional SSL/TLS certificate in `/config/ssl-certificate.p12`. Enables HTTPS ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#enabling-https)) |
| `MICRO_POSTGRESQL_PASSWORD`  | 4.0.0+  | The password for the optional PostgreSQL database ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#persisting-events-across-restarts))                              |
| `MICRO_AZURE_BLOB_ACCOUNT`   | 4.0.0+  | The Azure blob storage account name to use for downloading enrichment assets                                                                                                   |
| `MICRO_AZURE_BLOB_SAS_TOKEN` | 4.0.0+  | The Azure blob storage account token to use for downloading enrichment assets                                                                                                  |

---

# Snowplow Mini Control Plane API
> Control and configure a Snowplow Mini instance through the Control Plane API with HTTP authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/control-plane-api/

Snowplow Mini Control Plane API is created for controlling and configuring the Snowplow Mini instance without ssh to it.

You can use control plane from Snowplow Mini dashboard or you can send a request to a specific endpoint of the API with any HTTP client e.g. cURL

### Authentication

The Control Plane uses [HTTP basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).

This means that you need to add "-u username:password" to all `curl` commands, where `username` and `password` are as you specified in the Snowplow Mini setup.

### Current Methods

#### Service restart

```bash
/control-plane/restart-services
```

Example using `curl`:

```bash
$ curl -XPUT http://${snowplow_mini_ip}/control-plane/restart-services \       -u username:password
```

Restarts all the services running on the Snowplow Mini, including Stream Collector, Stream Enrich, Elasticsearch Loader.

This API call blocks until all the services have been restarted.

Return status 200 means that services have been successfully restarted.

#### Resetting Opensearch indices

As of 0.13.0, it is possible to reset Opensearch (or previously Elasticsearch) indices, along with the corresponding index patterns in Opensearch Dashboards, through Control Plane API.

```bash
curl -L \

-X POST '<mini-address>/control-plane/reset-service' \

-u '<username>:<password>' \

-H 'Content-Type: application/x-www-form-urlencoded' \

--data-urlencode 'service_name=elasticsearch'
```

Note that resetting deletes not only indices and patterns but also all events stored so far.

#### Restart services individually

As of 0.13.0, it is possible to restart services one by one.

```bash
curl -L \

-X PUT '<mini-address>/control-plane/restart-service' \

-u '<username>:<password>' \

-H 'Content-Type: application/x-www-form-urlencoded' \

--data-urlencode 'service_name=<service_name>'
```

where `service_name` can be one of the following: `collector`, `enrich`, `esLoaderGood`, `esLoaderBad`, `iglu`, `kibana`, `elasticsearch`.

#### Configuring telemetry

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information on telemetry.

HTTP GET to get current configuration

```bash
curl -L -X GET '<mini-address>/control-plane/telemetry' -u '<username>:<password>'
```

HTTP PUT to set it (use true or false as value of key `disable` to turn it on or off)

```bash
curl -L -X PUT '<mini-address>/control-plane/telemetry' -u '<username>:<password>' -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'disable=false'
```

#### Adding external Iglu Server

```bash
/control-plane/external-iglu
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/external-iglu \

  -d "uri=${external_iglu_uri}&apikey=${external_iglu_server_apikey}&vendor_prefix=${vendor_prefix}&name=${iglu_server_name}&priority=${priority}" \

  -u username:password
```

Adds given pieces of information of the external Iglu Server to iglu resolver json file to use it with Stream Enrich.

Note that a lower priority number means that the repo is ranked higher in terms of priority.

Return status 200 means that pieces of information are added to iglu resolver json file and Stream Enrich is restarted successfully.

**Note**: Apikey must follow the UUID format.

#### Uploading custom enrichments

```bash
/control-plane/enrichments
```

Example using `curl`:

```bash
curl http://${snowplow_mini_ip}/control-plane/enrichments \

  -F "enrichmentjson=@${path_of_the_custom_enrichment_dir}" \

  -u username:password
```

Upload custom enrichment json file to enrichments directory and restarts the Stream Enrich to activate uploaded custom enrichment.

Return status 200 means that custom enrichment json file is placed in the enrichments directory and Stream Enrich is restarted successfully.

#### Adding apikey for local Iglu Server

```bash
/control-plane/local-iglu-apikey
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/local-iglu-apikey \

  -d "local_iglu_apikey=${new_local_iglu_apikey}" \

  -u username:password
```

Adds provided apikey to the section of local Iglu Server in the iglu resolver json config. Restarts to Stream Enrich to activate the changes.

Return status 200 means that apikey is added and Stream Enrich is restarted successfully.

**Note**: Apikey must follow the UUID format.

#### Changing credentials for basic HTTP authentication

As of version 0.13.0, this endpoint doesn't accept new passwords shorter than 8 chars and with a score lower than 4 according to [zxcvbn](https://pkg.go.dev/github.com/trustelem/zxcvbn)

```bash
/control-plane/credentials
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/credentials \

  -d "new_username=${new_username}&new_password=${new_password}" \

  -u username:password
```

Changes the credentials for basic HTTP authentication.

You will get always empty reply from the server because caddy server will be restarted after credentials are submitted and the connection will be lost until caddy server is up and running.

#### Add domain name

```bash
/control-plane/domain-name
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/domain-name \

  -d "domain_name=${registered_domain_name}" \

  -u username:password
```

Adds domain name for Snowplow Mini instance. After adding the domain name, your connection will be secured with TLS automatically. Make sure that given domain name is resolving to Snowplow Mini instance IP address.

You will get always empty reply from the server because caddy server will be restarted after the domain name is submitted and the connection will be lost until caddy server is up and running.

#### Get Snowplow Mini version

```bash
/control-plane/version
```

Example using `curl`:

```bash
curl -XGET http://${snowplow_mini_ip}/control-plane/version \

  -u username:password
```

Returns version of the running Snowplow Mini instance.

#### Uploading Iglu Server configuration

```bash
/control-plane/iglu-config
```

Example using `curl`:

```bash
curl http://${snowplow_mini_ip}/control-plane/iglu-config \

  -F "igluserverhocon=@${path_of_the_iglu_server_config}" \

  -u username:password
```

Uploads Iglu Server configuration file and restarts the Iglu Server to activate uploaded configuration.

Return status 200 means that configuration is uploaded and Iglu Server is restarted successfully.

---

# Introduction to Snowplow Mini
> Snowplow Mini is a single-instance development environment for testing tracker updates and schema changes.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/

[Snowplow Mini](/docs/api-reference/snowplow-mini/) is a single-instance version of Snowplow that primarily serves as a development environment, giving you a quick way to debug tracker updates and changes to your schema and pipeline configuration.

> **Tip:** For new testing environments, we recommend using [Snowplow Micro](/docs/testing/snowplow-micro/), which you can [deploy through Console](/docs/testing/snowplow-micro/console/) or [run locally](/docs/testing/snowplow-micro/local/). New Snowplow Mini deployments are no longer available through Console.

You might use Snowplow Mini when:

- You've updated a schema in your Development environment and wish to send some test events against it before promoting it to Production
- You want to enable an Enrichment in a test environment before enabling it on Production

## Getting started

New Snowplow Mini instances are no longer available through Console. For new development environments, use [Snowplow Micro](/docs/testing/snowplow-micro/console/) instead.

For Snowplow Self-Hosted, see the setup guides for [AWS](/docs/api-reference/snowplow-mini/setup-guide-for-aws/) and [GCP](/docs/api-reference/snowplow-mini/setup-guide-for-gcp/).

## Conceptual diagram

![Architecture diagram showing Snowplow Mini (dev sandbox) alongside the Production Pipeline. Both receive events from Snowplow SDKs and webhooks via separate dev and prod tracker endpoints. Each pipeline runs Collector, Validate, and Enrich stages; events failing validation are separated out. Mini stores results in Elasticsearch good and bad indexes backed by a dev schema registry, while Production stores results in a data warehouse and bad logs backed by a prod schema registry.](/assets/images/image-72cd90387cfe692a867dd033688a5254.png)

The diagram above illustrates how Snowplow Mini (top) works alongside your Production pipeline (bottom).

By pointing your tracker(s) to the Collector on your Snowplow Mini you can send events from your applications development and QA environments to Snowplow Mini for testing.

Once you are happy with the changes you have made you would then change the trackers in your application to point to the Collector on your Production pipeline.[](https://github.com/snowplow/snowplow-mini#features)

## Features of Snowplow Mini

- Data is tracked and processed in real time
- Your Snowplow Mini speaks to your [Schema registries](/docs/fundamentals/schemas/#iglu-schema-repository) to allow events to be sent against your custom schemas
- Data is validated during processing
- Data is loaded into OpenSearch and can be queried directly or through the OpenSearch Dashboard
- Successfully processed events and failed events are in distinct good and bad indexes

## Topology

Snowplow-Mini runs several distinct applications on the same box which are all linked by NSQ topics. In a production deployment each instance could be an Autoscaling Group and each NSQ topic would be a distinct Kinesis Stream.

- Scala Stream Collector:

  - Starts server listening on `http://< sp mini public ip>/` which events can be sent to.
  - Sends "good" events to the `RawEvents` NSQ topic
  - Sends "bad" events to the `BadEvents` NSQ topic

- Stream Enrich:

  - Reads events in from the `RawEvents` NSQ topic
  - Sends events which passed the enrichment process to the `EnrichedEvents` NSQ topic
  - Sends events which failed the enrichment process to the `BadEvents` NSQ topic

- OpenSearch Sink Good:

  - Reads events from the `EnrichedEvents` NSQ topic
  - Sends those events to the `good` OpenSearch index
  - On failure to insert, writes errors to `BadElasticsearchEvents` NSQ topic

- OpenSearch Sink Bad:

  - Reads events from the `BadEvents` NSQ topic
  - Sends those events to the `bad` OpenSearch index
  - On failure to insert, writes errors to `BadElasticsearchEvents` NSQ topic

These events can then be viewed in Kibana at `http://< sp mini public ip>/kibana`.

---

# Set up Snowplow Mini on AWS
> Deploy Snowplow Mini on AWS for a single-instance testing environment.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/setup-guide-for-aws/

Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:

1. Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
2. Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
3. Giving people running Snowplow a quick way to debug tracker updates

All setup for Snowplow Mini is done within the AWS Console and will incur small amounts of running costs, depending on the size of the EC2 instance you select.

We offer Snowplow Mini in 3 different sizes. To decide on which size of Snowplow Mini to choose, read on.

## large & xlarge & xxlarge

Mini is available in 3 different sizes:

- `large` : Opensearch has `4g` heap size and Snowplow apps has `0.5g` heap size. Recommended machine RAM is `8g`.
- `xlarge` : Double the large image. Opensearch has `8g` heap size and Snowplow apps has `1.5g` heap size. Recommended machine RAM is `16g`.
- `xxlarge` : Double the xlarge image. Opensearch has `16g` heap size and Snowplow apps has `3g` heap size. Recommended machine RAM is `32g`.

This service is available as an EC2 image within the AWS Community AMIs in the following regions: `ap-northeast-1`, `ap-northeast-2`, `ap-south-1`, `ap-southeast-1`, `ap-southeast-2`, `ca-central-1`, `eu-central-1`, `eu-west-1`, `eu-west-2`, `sa-east-1`, `us-east-1`, `us-east-2`, `us-west-1` and `us-west-2`.

Version 0.26.0 (recommended) comes with:

- Snowplow Collector NSQ 3.7.0
- Snowplow Enrich NSQ 6.11.0
- Snowplow Elasticsearch Loader 2.1.3
- Snowplow Iglu Server 0.14.0
- Opensearch 3.6.0
- Opensearch Dashboards 3.6.0
- Postgresql 16.10
- NSQ v1.3.0

Note: All services are configured to start automatically so everything should happily survive restarts/shutdowns.

To understand the flow of data please refer to the following diagram:

![Real-time processing topology diagram showing the Scala Stream Collector, Stream Enrich, Elasticsearch Sink (Good), and Elasticsearch Sink (Bad) services interacting with RawEvents, EnrichedEvents, BadEvents, and BadElasticsearchEvents streams, with both Elasticsearch Sinks writing to an Elasticsearch node](/assets/images/snowplow-mini-topology-95da73899375d477bfe132b2bcdb0e19.jpg)

**IAM**

Create a role with the following configuration

- Step 1: For `Select type of trusted entity` , select `EC2`
- Step 2.1: For `Attach permissions policies` , create a policy with the following

```json
{

  "Version" : "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "s3:GetObject",

        "logs:CreateLogStream",

        "logs:PutLogEvents"

      ],

      "Resource": ["*"]

    }

  ]

}
```

- Step 2.2: In step 2 of role creation, select the policy created in the previous step
- Step 3: Tags are optional
- Step 4: Fill in the role name and create it.

**CloudWatch**

Create a log group named `snowplow-mini` so that Mini can emit logs to this log group.

Mini will not function properly if a log group with that name isn't found.

## Security Group

In the EC2 Console UI select `Security Groups` from the panel on the left.

Select the `Create Security Group` button and fill in the name, description and what VPC you want to attach it to.

You will then need to add the following InBound rules:

![The AWS EC2 \&quot;Edit inbound rules\&quot; dialog showing three Custom TCP rules: port 80 open to 0.0.0.0/0, port 443 open to 0.0.0.0/0, and port 22 restricted to a specific IP address](/assets/images/security-groups-setup-f8d299c2b2b111f0dbd2351e83ec119e.png)

- Custom TCP Rule | Port Range (80)

- CIDR range `0.0.0.0/0`

- Custom TCP Rule | Port Range (443)

- CIDR range `0.0.0.0/0`

- SSH (optional):

  - Custom TCP Rule | Port Range (22)
  - CIDR range `{{ YOUR IP HERE }}/32`

For OutBound you can leave the default to allow everything out.

## Choose AMI

In the EC2 Console UI select the `Launch Instance` button then select the `Community AMIs` button. In the search bar enter `snowplow-mini-0.26.0` to find the needed AMI and then select it.

## Choose Instance Type

AMI names explicitly specify which instance type to use.

- `0.26.0-large` needs `t2.large`
- `0.26.0-xlarge` needs `t2.xlarge`
- `0.26.0-xxlarge` needs `t2.2xlarge`

## Configure Instance

- Select the IAM role created above.

- If you created your Security Group in a different VPC than the default you will need to select the same VPC in the Network field.

**NOTE**: If you select a custom VPC ensure that you select `Enable` for the Auto-assign Public IP option.

## Add Storage

Depending on how long you intend to run Snowplow Mini and how much data you intend to send/store you will need to change the size of the block store accordingly.

For basic testing and debugging;

- 20-50 Gb should suffice for `large`
- 50-100 Gb should suffice for `xlarge`
- 100-200 Gb should suffice for `xxlarge`

We also recommend changing the `Volume Type` to GP2 from Magnetic for a smoother experience.

## Tag Instance

Add any tags you like here.

## Configure Security Group

Select the Security Group you created [above](#security-group).

## Review

Press the `Launch` button and select an existing key-pair, or create a new one, if you want to be able to SSH into the box.

**Telemetry notice**

By default, Snowplow collects telemetry data for Mini (since version 0.13.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to disable telemetry, you can do so via the [API](/docs/api-reference/snowplow-mini/control-plane-api/#configuring-telemetry).

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Set up Snowplow Mini on GCP
> Deploy Snowplow Mini on GCP for a single-instance testing environment.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/setup-guide-for-gcp/

Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:

1. Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
2. Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
3. Giving people running Snowplow a quick way to debug tracker updates (because they can)

Version 0.26.0 (recommended) comes with:

- Snowplow Collector NSQ 3.7.0
- Snowplow Enrich NSQ 6.11.0
- Snowplow Elasticsearch Loader 2.1.3
- Snowplow Iglu Server 0.14.0
- Opensearch 3.6.0
- Opensearch Dashboards 3.6.0
- Postgresql 16.10
- NSQ v1.3.0

Note: All services are configured to start automatically so everything should happily survive restarts/shutdowns.

To understand the flow of data please refer to the following diagram:

![Data flow diagram showing the Snowplow Mini real-time processing topology: the Scala Stream Collector receives events, passes them to Stream Enrich, and both good and bad events flow through Elasticsearch Sink components into an Elasticsearch node, with four named queues — RawEvents, EnrichedEvents, BadEvents, and BadElasticsearchEvents — shown as horizontal streams between components.](/assets/images/snowplow-mini-topology-95da73899375d477bfe132b2bcdb0e19.jpg)

## Importing public tarballs to a GCP project

Our offering for GCP is 3 compressed tarballs for 3 different sized Snowplow Mini, produced through `gcloud`'s [`export`](https://cloud.google.com/sdk/gcloud/reference/compute/images/export) command. Simply put, they are Virtual Disk images exported in GCP format, a `disk.raw` file that has been tarred and gzipped.

To use them within GCP console, you need to import a tarball of your choice into your GCP project. You can use `gcloud` CLI utility to do that.

Browse [GCP docs](https://cloud.google.com/sdk/docs/quickstarts) to get started with `gcloud`.

Assuming you have `gcloud` setup and configured for your GCP project, use `gcloud`'s [`create`](https://cloud.google.com/sdk/gcloud/reference/compute/images/create) command to import a tarball of your choice into your GCP project.

A sample usage would be as following.

```bash
gcloud compute images create \

imported-sp-mini \

--source-uri \

https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-26-0-large-1780921141.tar.gz
```

Note that `imported-sp-mini` is a name of your choice for destination image and above URI is for large image, change it with your preferred version of Snowplow Mini.

Version 0.26.0 (recommended)

| L / 2 vCPUs                                                                                        | XL / 4 vCPUs                                                                                         | XXL / 8 vCPUs                                                                                          |
| -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| [large](https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-26-0-large-1780921141.tar.gz) | [xlarge](https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-26-0-xlarge-1780921124.tar.gz) | [xxlarge](https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-26-0-xxlarge-1780921136.tar.gz) |

You can find more about `gcloud compute images create` command [here](https://cloud.google.com/sdk/gcloud/reference/compute/images/create) for additional parameters.

After importing our tarball of your choice into your project, you should see it under `Images` on `Compute Engine`.

To decide on which size of Snowplow Mini to choose, read on.

## large & xlarge & xxlarge

Mini is available in 3 different sizes:

- `large` : Opensearch has `4g` heap size and Snowplow apps has `0.5g` heap size. Recommended machine RAM is `8g`.
- `xlarge` : Double the large image. Opensearch has `8g` heap size and Snowplow apps has `1.5g` heap size. Recommended machine RAM is `16g`.
- `xxlarge` : Double the xlarge image. Opensearch has `16g` heap size and Snowplow apps has `3g` heap size. Recommended machine RAM is `32g`.

## Create instance

Go to `Compute Engine` on GCP console, select `Images` from menu on the left. You should see your imported image on the list. Select it then you should see `CREATE INSTANCE` button at the top of the page. Click on it.

![GCP Console Images page showing the imported Snowplow Mini image selected, with the CREATE INSTANCE button visible in the top action bar.](/assets/images/create-instance-cc7dee2edb679cbcfab716d3f068aa49.png)

![GCP Create an instance form showing the instance name set to \&quot;snowplow-mini-0-6-0\&quot;, region us-east1 (South Carolina), zone us-east1-b, and machine type configured with 2 vCPUs and 8 GB memory.](/assets/images/create-instance-2-b23317e188861463d0a697e7e6698441.png)

![Lower section of the GCP Create an instance form showing boot disk set to a 10 GB standard persistent disk with the Snowplow Mini image, the Compute Engine default service account selected with Allow default access scope, HTTP traffic allowed in the Firewall section, and the Create button at the bottom.](/assets/images/create-instance-3-7397ccd9528d0226acb3bf709bdccf5a.png)

Click `Create`.

These images show setup for `large` image. To setup `xlarge` or `xxlarge`, you should increase memory per above explanation of different sizes.

**Telemetry notice**

By default, Snowplow collects telemetry data for Mini (since version 0.13.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to disable telemetry, you can do so via the [API](/docs/api-reference/snowplow-mini/control-plane-api/#configuring-telemetry).

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Snowplow Mini usage guide
> Learn how to use Snowplow Mini for testing, debugging trackers, and exploring Snowplow features.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/usage-guide/

## Overview

Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:

1. Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
2. Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
3. Giving people running Snowplow a quick way to debug tracker updates

Jump to [First Time Usage](#first-time-usage) if it is your first time with a Mini.

## Upgrading

Until version 0.15.0, Snowplow data was loaded to Elasticsearch 6.x in the Mini. However, a [licensing change](https://www.elastic.co/blog/licensing-change) in Elasticsearch prevented us from upgrading it to more recent versions. To make sure we stay up to date with important security fixes, we've decided to replace Elasticsearch with [Opensearch](https://opensearch.org/). Also, Kibana is replaced with [Opensearch Dashboards](https://opensearch.org/docs/latest/dashboards/index/).

[Opensearch](https://opensearch.org/) is a fork of open source Elasticsearch 7.10. Therefore this change shouldn't affect Mini users much. To minimize the impact further, we've tried to make as minimal a change as possible. In Mini, you can still access Opensearch with the `/elasticsearch` endpoint and Opensearch Dashboards with the `/kibana` endpoint.

The only breaking change this migration would bring is the removal of mapping types. It means that you don't have to provide a mapping type in your search queries anymore when accessing your data in good or bad indices. For example, good events count can be found using the following endpoint in previous versions: `/elasticsearch/good/good/_count`. Starting with 0.15.0, it can be found using this endpoint: `/elasticsearch/good/_count`.

## First time usage

This section is dedicated to the steps that need performing when accessing the Snowplow Mini instance for the first time.

### Connecting to the instance for the first time

You can access the Snowplow Mini instance at the `http://[public_dns]/home` address. While accessing Snowplow Mini services, HTTP authentication is required. As a result, you will be prompted for credentials which are `USERNAME_PLACEHOLDER` and `PASSWORD_PLACEHOLDER` by default.

You **should** change these default credentials to something to your liking by going to the Control Plane tab (the last one) and fill the "Change username and password for basic http authentication" form towards the bottom.

**Note that only alphanumeric passwords are supported.**

You will then be prompted for those new credentials.

### Changing the super API key for the local Iglu schema registry

As as second step, you should change the super API key for the Iglu schema registry that is bundled with Snowplow Mini. This API key can be changed via the Control Plane tab.

Given that this API key must be a UUID v4, you will need to generate one by running `uuidgen` at the command line, or by using an online UUID generator like [this one](https://www.uuidgenerator.net/). Make a note of this UUID, you'll need it to upload your own event and context schemas to Snowplow Mini in the next subsection.

### Generating a pair of read/write API keys for the local Iglu schema registry

> **Note:** Mini 0.8.0 comes bundled with Iglu Server 0.6.1 which introduced a couple of changes regarding this section.
>
> - Swagger UI of Iglu Server is deprecated, however Iglu Server still serves at `/iglu-server` endpoint.
> - `POST /api/auth/keygen` no longer supports query parameter to provide vendor prefix. Use POST raw data request instead.

To add schemas to the Iglu repository bundled with Snowplow Mini, you have to create a dedicated pair of API keys. There are 2 options:

- Use igluctl’s [server keygen](/docs/api-reference/iglu/igluctl/#server-keygen) subcommand
- Use any HTTP client e.g. cURL

#### Option 1

First, [download igluctl](/docs/api-reference/iglu/igluctl/#downloading-and-running-igluctl).

Following is a sample execution where `com.acme` is the vendor prefix for which we'll upload our schemas, `mini-address` is the URL of our mini and `53b4c441-84f7-467e-af4c-074ced53eb20` is an example super API key you would have created in the previous steps.

```bash
/path/to/igluctl server keygen --vendor-prefix com.acme mini-address/iglu-server 53b4c441-84f7-467e-af4c-074ced53eb20
```

#### Option 2

You can also use `cURL` to interact with Iglu Server:

```text
curl --location --request POST 'mini-address/iglu-server/api/auth/keygen' \

  --header 'apikey: 1b5d0459-3492-451c-aab1-7f74cbe12112' \

  --header 'Content-Type: application/json' \

  --data-raw '{"vendorPrefix":"com.acme"}'
```

should return a read key and a write key.

```json
{

  "read":"bfa90866-ab14-4b92-b6ef-d421fd688b54",

  "write":"6175aa41-d3a7-4e4f-9fb4-3a170f3c6c16"

}
```

### Copying your Iglu repository to Snowplow Mini (optional)

To test and send non-standard Snowplow events such as your own custom-contexts and unstructured events you can load them into the Iglu repository local to the Snowplow Mini instance.

1. Get a local copy of your Iglu repository which contains your schemas. This should be modelled after [this folder](https://github.com/snowplow/iglu-central/tree/master/schemas)

2. [Download igluctl](/docs/api-reference/iglu/igluctl/#downloading-and-running-igluctl).

3. Run the executable with the following input:

- The address of the Iglu repository: `http://[public_dns]/iglu-server`
- The Super API Key you created previously
- The path to your schemas For example to load the `iglu-central` repository into Iglu Server:

```bash
/path/to/igluctl static push iglu-central/schemas http://[public_dns]/iglu-server 980ae3ab-3aba-4ffe-a3c2-3b2e24e2ffce --public
```

Note: this example assumes the `iglu-central` repository has been cloned in the same directory as where executable is run.

1. After uploading the schemas, you will need to clear the cache with the restart button under the Control Plane tab in the Snowplow Mini dashboard.

### Setting up HTTPS (optional)

If you want to use HTTPS to connect to Snowplow Mini, you need to submit a domain name via the Control Plane. Make sure that the domain name you submit is redirected to the IP of the server Snowplow Mini is running from.

## Sending events to Snowplow Mini

Now that the first time usage steps have been dealt with, you can send some events!

### Example events

An easy way to quickly send a few test events is to use our example web page.

1. Open up the Snowplow Mini UI at: `http://[public_dns]/home`
2. Login with username and password which you choose in step 2.1
3. Select the `Example Events` tab
4. Press the event triggering buttons on the page!

### Events from tracker

You can instrument any other Snowplow tracker by specifying the collector URL as the public DNS of the Snowplow Mini instance.

## Accessing the Opensearch API

Snowplow Mini makes the Opensearch (or previously Elasticsearch) HTTP API available at `http://[public_dns]/elasticsearch`, you can check it's working by:

- Checking the Opensearch API is available:

  - `curl --user username:password http://[public_dns]/elasticsearch`
  - You should see a `200 OK` response

- Checking the number of good events we sent in step 3:

  - `curl --user username:password http://[public_dns]/elasticsearch/good/_count`
  - You should see the appropriate count of sent events

## Viewing the data in Opensearch Dashboards

Data sent to Snowplow Mini will be processed and loaded into Opensearch in real time. In turn, it will be made available in Opensearch Dashboards. To view the data in Opensearch Dashboards, navigate in your browser to `mini-public-address/kibana`.

### Index patterns

Snowplow Mini comes with two index patterns:

- `good` : For good events, indexed on `collector_tstamp`
- `bad` : For bad events, indexed in `data.failure.timestamp`

### Discover your data

Browse to `mini-public-address/kibana` , once Opensearch Dashboards is loaded, you should be able to view most recently sent good events via the discover interface:

You can then inspect any individual event data in the UI by unfolding a payload:

![Opensearch Dashboards Discover interface showing 12 good events in the \&quot;good\&quot; index, with available fields listed on the left and a table of event rows on the right displaying timestamps and source data including fields such as collector\_tstamp, v\_collector, and br\_features.](/assets/images/Screen-Shot-2020-04-13-at-13.20.22-8410d743b7a7a1261de9528d561c8aa7.jpg)

If you want to inspect bad events, click on `good`, placed towards top left of the screen and select `bad` from drop-down list.

![Opensearch Dashboards Discover interface showing 13 bad events in the \&quot;bad\&quot; index, with failure-related fields listed on the left (including data.failure.messages, data.failure.timestamp, data.payload.body) and event rows on the right showing error details such as \&quot;empty querystring\&quot; and \&quot;not a valid URI redirect\&quot;.](/assets/images/Screen-Shot-2020-04-13-at-13.32.26-1b9f1eaae978503d3f587585563148df.jpg)

Unfold any payload to inspect a bad event in detail.

![Opensearch Dashboards Discover interface showing a single good event unfolded, with its full JSON payload displayed in a table view including fields such as \_index, app\_id, geo\_latitude, v\_collector, and collector\_tstamp.](/assets/images/Screen-Shot-2020-04-13-at-13.23.16-970d83b883ef507d69792cf0f65de9eb.jpg)

## Resetting Opensearch indices

As of 0.13.0, it is possible to reset Opensearch (or previously Elasticsearch) indices, along with the corresponding index patterns in Opensearch Dashboards, through Control Plane API.

```bash
curl -L \

-X POST '<mini-address>/control-plane/reset-service' \

-u '<username>:<password>' \

-H 'Content-Type: application/x-www-form-urlencoded' \

--data-urlencode 'service_name=elasticsearch'
```

Note that resetting deletes not only indices and patterns but also all events stored so far.

## Restart services individually

As of 0.13.0, it is possible to restart services one by one.

```bash
curl -L \

-X PUT '<mini-address>/control-plane/restart-service' \

-u '<username>:<password>' \

-H 'Content-Type: application/x-www-form-urlencoded' \

--data-urlencode 'service_name=<service_name>'
```

where `service_name` can be one of the following: `collector`, `enrich`, `esLoaderGood`, `esLoaderBad`, `iglu`, `kibana`, `elasticsearch`.

## Configuring telemetry

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information on telemetry.

HTTP GET to get current configuration

```bash
curl -L -X GET '<mini-address>/control-plane/telemetry' -u '<username>:<password>'
```

HTTP PUT to set it (use true or false as value of key `disable` to turn it on or off)

```bash
curl -L -X PUT '<mini-address>/control-plane/telemetry' -u '<username>:<password>' -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'disable=false'
```

## Uploading custom enrichments

You can add new custom enrichments via the Control Plane tab. The only thing you have to do is submit the enrichment configuration file which you created according to the documentation in [Available Enrichments](/docs/pipeline/enrichments/available-enrichments/). If the enrichment relies on additional schemas these should be uploaded to the Iglu repository.

## Adding a custom schema

Since Mini 0.8.0 deprecated Swagger UI of Iglu Server, we have 2 options:

- Use igluctl’s [static push](/docs/api-reference/iglu/igluctl/#static-push) subcommand to put our custom schema into the Iglu Server
- Use any HTTP client e.g. cURL

#### Option 1

First, [download igluctl](/docs/api-reference/iglu/igluctl/#downloading-and-running-igluctl).

Following is a sample execution where `path-to-schema(s)` is the path to custom schema(s) , `mini-address` is the URL of our mini and `53b4c441-84f7-467e-af4c-074ced53eb20` is an example super API key you would have created in the previous steps.

```bash
/path/to/igluctl static push path-to-schema(s) mini-address/iglu-server 53b4c441-84f7-467e-af4c-074ced53eb20
```

#### Option 2

You can also use `cURL` to interact with Iglu Server:

```bash
curl mini-address/iglu-server/api/schemas -X POST \

  -H "apikey: YOUR_APIKEY" -d '{"json": YOUR_JSON}'
```

The command will produce a response like this one, if no errors are encountered:

```json
{

  "message": "Schema created",

  "updated": false,

  "location": "iglu:com.acme/ad_click/jsonschema/1-0-0",

  "status":201

}
```

## Adding an external Iglu repository

If you already have an external Iglu repository available, instead of copying it inside the Iglu repository bundled with the Snowplow Mini instance as shown in 2.4, you can add it directly with the Control Plane's `Add an external Iglu repository` form. Note that if you're using a static repository hosted on S3, you can omit providing an API key.

## Runtime metrics

Mini 0.12.0 introduced /metrics endpoint powered by [cAdvisor](https://github.com/google/cadvisor) . You can also find the link to metrics on the home page under Quicklinks header.

It's been possible to observe runtime metrics of a Mini instance by looking at AWS/GCP monitoring dashboards however internal services' individual metrics weren't exposed, making it more difficult to diagnose issues.

Exposing runtime metrics such as CPU, RAM and Network usage of the internal services in real time will make Mini more transparent, hopefully making it easier to understand what's going on under the hood.

## Logs

As of Mini 0.12.0, application logs of the Mini sub-services are exported to Cloudwatch on AWS and Cloud Logging on GCP.

On AWS, each individual service emits its' logs under a specific log stream within `snowplow-mini` log group.

On GCP, we need to make use of filters to see the logs of a specific component. The recommended approach is as following:

- On GCP console, go to Logging > Logs Viewer
- Under Query Builder, select resource
- Under `VM instance`, select the instance Mini is running at
- Click on `Add`

Click on `Run Query` and we should see logs of all services combined.

To see the logs of a specific component, add the following filter to the query:

jsonPayload.container.name="/service-name"

where service-name can be one of the following: `elasticsearch`, `kibana`, `elasticsearch-loader-good`, `elasticsearch-loader-bad`, `nsqlookupd`, `nsqd`, `nsqadmin`, `scala-stream-collector-nsq`, `stream-enrich-nsq`

An example query looks as following:

resource.type="gce\_instance"

resource.labels.instance\_id="3778299199368430127"

jsonPayload.container.name="/elasticsearch"

---

# Collector configuration reference
> Complete configuration reference for the Collector HOCON config file, including common options, sink-specific settings, cookie management, networking, and TLS configuration.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/configure/

This is a complete list of the options that can be configured in the collector HOCON config file. The [example configs in github](https://github.com/snowplow/stream-collector/tree/master/examples) show how to prepare an input file. Some features are described in more detail at the bottom of this page.

### License

The collector is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the collector, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `collector.license.accept` option, like this:

```hcl
collector {

  license { accept = true }

}
```

### Common options

| parameter                                                                         | description                                                                                                                                                                                                                                                                                                                                                     |
| --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.interface`                                                             | Required. E.g. `0.0.0.0`. The collector listens for http requests on this interface.                                                                                                                                                                                                                                                                            |
| `collector.port`                                                                  | Required. E.g. `80`. The collector listens for http requests on this port.                                                                                                                                                                                                                                                                                      |
| `collector.ssl.enable`                                                            | Optional. Default: `false`. The collector will also listen for https requests on a different port.                                                                                                                                                                                                                                                              |
| `collector.ssl.port`                                                              | Optional. Default: `443`. The port on which to listen for https requests.                                                                                                                                                                                                                                                                                       |
| `collector.ssl.redirect`                                                          | Optional. Default: `false`. If enabled, the collector redirects http requests to the https endpoint using a `301` status code.                                                                                                                                                                                                                                  |
| `collector.hsts.enable` _(since 3.1.0)_                                           | Default: `false`. Whether to send an [HSTS header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security).                                                                                                                                                                                                                        |
| `collector.hsts.maxAge` _(since 3.1.0)_                                           | Default: `365 days`. The maximum age for the HSTS header.                                                                                                                                                                                                                                                                                                       |
| `collector.paths`                                                                 | Optional. More details about this feature below. This is for customising the collector's endpoints. You can also map any valid (ie, two-segment) path to one of the three default paths.                                                                                                                                                                        |
| `collector.p3p.policyRef`                                                         | Optional. Default: `/w3c/p3p.xml`. Configures the p3p http header.                                                                                                                                                                                                                                                                                              |
| `collector.p3p.CP`                                                                | Optional. Default: `NOI DSP COR NID PSA OUR IND COM NAV STA`. Configures the p3p http header.                                                                                                                                                                                                                                                                   |
| `collector.crossDomain.enabled`                                                   | Optional. Default: `false`. If enabled, `/crossdomain.xml` endpoint returns a cross domain policy file.                                                                                                                                                                                                                                                         |
| `collector.crossDomain.domains`                                                   | Optional. Default: `[*]` meaning the cross domain policy file allows all domains. This can also be changed to a list of domains.                                                                                                                                                                                                                                |
| `collector.crossDomain.secure`                                                    | Optional. Default: `true`. Whether cross domain policy file grants access to just HTTPS or both HTTP and HTTPS sources                                                                                                                                                                                                                                          |
| `collector.cookie.enabled`                                                        | Optional. Default: `true`. The collector sets a cookie to set the user's network user id. Changing this to `false` disables setting cookies. Regardless of this setting, if the collector receives a request with the custom `SP-Anonymous:*` header, no cookie will be set. You can control whether this header is set or not in your tracking implementation. |
| `collector.cookie.expiration`                                                     | Optional. Default: `365 days`. Expiry of the collector's cookie.                                                                                                                                                                                                                                                                                                |
| `collector.cookie.name`                                                           | Optional. Default: `sp`. Name of the collector's cookie.                                                                                                                                                                                                                                                                                                        |
| `collector.cookie.domains`                                                        | Optional. Default: no domains. There is more details about this feature below. This is for fine control over the cookie's domain attribute.                                                                                                                                                                                                                     |
| `collector.cookie.fallbackDomain`                                                 | Optional. If set, the fallback domain will be used for the cookie if none of the `Origin` header hosts matches the list of cookie domains.                                                                                                                                                                                                                      |
| `collector.cookie.secure`                                                         | Optional. Default: `true`. Sets the `secure` property of the cookie.                                                                                                                                                                                                                                                                                            |
| `collector.cookie.httpOnly`                                                       | Optional. Default: `true`. Sets the `httpOnly` property of the cookie. We recommend `true` because `httpOnly` cookies are allowed a longer expiry time by web browsers.                                                                                                                                                                                         |
| `collector.cookie.sameSite`                                                       | Optional. Default: `None`. Sets the `sameSite` property of the cookie. Possible values: `Strict`, `Lax`, `None`.                                                                                                                                                                                                                                                |
| `collector.cookie.clientCookieName` (since 3.4.0)                                 | Optional. Default: not set. If a name is specified (e.g. `sp_client`), the collector sets an extra cookie with that name, `httpOnly=false` and the same value as the main cookie (network user id). This is useful if you need to access the network user id on the client side using JavaScript.                                                               |
| `collector.doNotTrackCookie.enabled`                                              | Optional. Default: `false`. If enabled, the collector respects a "do not track" cookie. If the cookie is present, it returns a `200` status code but it does not log the request to the output queue.                                                                                                                                                           |
| `collector.doNotTrackCookie.name`                                                 | Required when `doNotTrackCookie` feature is enabled. Configures the name of the cookie in which to check if tracking is disabled.                                                                                                                                                                                                                               |
| `collector.doNotTrackCookie.value`                                                | Required when `doNotTrackCookie` feature is enabled. Can be a regular expression. The value of the cookie must match this expression in order for the collector to respect the cookie.                                                                                                                                                                          |
| `collector.cookieBounce.enabled`                                                  | Optional. Default: `false`. When enabled, when the cookie is missing, the collector performs a redirect to itself to check if third-party cookies are blocked using the specified name. If they are indeed blocked, `fallbackNetworkId` is used instead of generating a new random one.                                                                         |
| `collector.cookieBounce.name`                                                     | Optional. Default: `n3pc`. Name of the request parameter which will be used on redirects checking that the third-party cookies work.                                                                                                                                                                                                                            |
| `collector.cookieBounce.fallbackNetworkUserId`                                    | Optional. Default: `00000000-0000-4000-A000-000000000000`. Network user id to use when third-party cookies are blocked.                                                                                                                                                                                                                                         |
| `collector.cookieBounce.forwardedProtocolHeader`                                  | Optional. E.g. `X-Forwarded-Proto`. The header containing the originating protocol for use in the bounce redirect location. Use this if behind a load balancer that performs SSL termination.                                                                                                                                                                   |
| `collector.enableDefaultRedirect`                                                 | Optional. Default: `false`. When enabled, the collector's `/r` endpoint returns a `302` status code with a redirect back to a url specified with the `?u=` query parameter.                                                                                                                                                                                     |
| `collector.redirectDomains` (since _2.5.0_)                                       | Optional. Default: empty. Domains which are valid for collector redirects. If empty then redirects are allowed to any domain. Must be an exact match.                                                                                                                                                                                                           |
| `collector.redirectMacro.enabled`                                                 | Optional. Default: `false`. When enabled, the redirect url passed via the `u` query parameter is scanned for a `placeholder` token. All occurrences of the placeholder are substituted with the cookie's network user id.                                                                                                                                       |
| `collector.redirectMacro.placeholder`                                             | Optional. Default: `${SP_NUID}`.                                                                                                                                                                                                                                                                                                                                |
| `collector.rootResponse.enabled`                                                  | Optional. Default: `false`. Enable custom response handling for the root `"/"` path.                                                                                                                                                                                                                                                                            |
| `collector.rootResponse.statusCode`                                               | Optional. Default: `302`. The http status code to use when root response is enabled.                                                                                                                                                                                                                                                                            |
| `collector.rootResponse.headers`                                                  | Optional. A map of key value pairs to include in the root response headers.                                                                                                                                                                                                                                                                                     |
| `collector.rootResponse.body`                                                     | Optional. The http response body to use when root response is enabled.                                                                                                                                                                                                                                                                                          |
| `collector.cors.accessControlMaxAge`                                              | Optional. Default: `60 minutes`. Configures how long a the results of a preflight request can be cached by the browser. `-1` seconds disables the cache.                                                                                                                                                                                                        |
| `collector.preTerminationPeriod` (since _2.5.0_)                                  | Optional. Default: `10 seconds`. Configures how long the collector should pause after receiving a sigterm before starting the graceful shutdown. During this period the collector continues to accept new connections and respond to requests.                                                                                                                  |
| `collector.prometheusMetrics.enabled` (deprecated since _2.6.0_)                  | Optional. Default: `false`. When enabled, all requests are logged as prometheus metrics and the `/metrics` endpoint returns the report about the metrics.                                                                                                                                                                                                       |
| `collector.prometheusMetrics.durationBucketsInSeconds` (deprecated since _2.6.0_) | Optional. E.g. `[0.1, 3, 10]`. Custom buckets for the `http_request_duration_seconds_bucket` duration prometheus metric.                                                                                                                                                                                                                                        |
| `collector.telemetry.disable`                                                     | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                       |
| `collector.telemetry.userProvidedId`                                              | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                             |
| `collector.compression.enabled` (since _3.6.0_)                                   | Optional. Default: `false`. Enable compression on the output. Compression should only be enabled with Enrich >=6.1.0.                                                                                                                                                                                                                                           |
| `collector.compression.type` (since _3.6.0_)                                      | Optional. Default: `zstd`. Compression algorithm to use.                                                                                                                                                                                                                                                                                                        |
| `collector.compression.gzipCompressionLevel` (since _3.6.0_)                      | Optional. Default: `6`. The compression level for GZIP compression. It is between 1 and 9. Lower levels have faster compression speed, but worse compression ratio.                                                                                                                                                                                             |
| `collector.compression.zstdCompressionLevel` (since _3.6.0_)                      | Optional. Default: `9`. The compression level for ZSTD compression. It is between 1 and 22. Lower levels have faster compression speed, but worse compression ratio.                                                                                                                                                                                            |

### Kinesis collector options

| parameter                                                                 | description                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                                                  | Required. Name of the output kinesis stream for successfully collected events.                                                                                                                                                                                                                                                        |
| `collector.streams.bad`                                                   | Required. Name of the output kinesis stream for http requests which could not be written to the good stream. For example, if the event size exceeds the kinesis limit of 1MB.                                                                                                                                                         |
| `collector.streams.useIpAddressAsPartitionKey` (deprecated since _3.5.0_) | Optional. Default: `false`. Whether to use the user's IP address as the kinesis partition key.                                                                                                                                                                                                                                        |
| `collector.streams.{good,bad}.region`                                     | Optional. Default: `eu-central-1`. AWS region of the kinesis streams.                                                                                                                                                                                                                                                                 |
| `collector.streams.{good,bad}.customEndpoint`                             | Optional. Override aws kinesis endpoint. Can be helpful when using localstack for testing.                                                                                                                                                                                                                                            |
| `collector.streams.{good,bad}.threadPoolSize`                             | Optional. Default: `10`. Thread pool size used by the collector sink for asynchronous operations.                                                                                                                                                                                                                                     |
| `collector.streams.{good,bad}.sqsMaxBytes`                                | Optional. Set to the name of a SQS topic to enable buffering of good output events. When messages cannot be sent to Kinesis, (e.g. because of exceeding api limits) then they get sent to SQS as a fallback. Helpful for smoothing over traffic spikes.                                                                               |
| `collector.streams.good.sqsGoodBuffer`                                    | Optional. Set to the name of a SQS topic to enable buffering of good output events. When messages cannot be sent to Kinesis, (e.g. because of exceeding api limits) then they get sent to SQS as a fallback. Helpful for smoothing over traffic spikes.                                                                               |
| `collector.streams.bad.sqsBadBuffer`                                      | Optional. Like the `sqsGoodBuffer` but for failed events.                                                                                                                                                                                                                                                                             |
| `collector.streams.{good,bad}.aws.accessKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_ACCESS_KEY_ID` environment variable.                                                                                                                                                                     |
| `collector.streams.{good,bad}.aws.secretKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_SECRET_ACCESS_KEY` environment variable.                                                                                                                                                                 |
| `collector.streams.{good,bad}.maxBytes` (since _2.9.0_)                   | Optional. Default: `1000000` (1 MB). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead. If SQS buffer is activated, `sqsMaxBytes` is used instead.                                                                                                    |
| `collector.streams.{good,bad}.sqsMaxBytes`                                | Optional. Default: `192000` (192 kb). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead. SQS has a record size limit of 256 kb, but records are encoded with Base64, which adds approximately 33% of the size, so we set the limit to `256 kb * 3/4`. |
| `collector.streams.{good,bad}.startupCheckInterval` (since _2.9.0_)       | Optional. Default: `1 second`. When collector starts, it checks if Kinesis streams exist with `describeStreamSummary` and if SQS buffers exist with `getQueueUrl` (if configured). This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted.        |
| `collector.streams.backoffPolicy.minBackoff`                              | Optional. Default: `3000`. Time (in milliseconds) for retrying sending to kinesis / SQS after failure.                                                                                                                                                                                                                                |
| `collector.streams.backoffPolicy.maxBackoff`                              | Optional. Default: `600000`. Time (in milliseconds) for retrying sending to kinesis / SQS after failure.                                                                                                                                                                                                                              |
| `collector.streams.buffer.byteLimit`                                      | Optional. Default: `3145728`. Incoming events are stored in an internal buffer before being sent to Kinesis. This configures the maximum total size of pending events.                                                                                                                                                                |
| `collector.streams.buffer.recordLimit`                                    | Optional. Default: `500`. Configures the maximum number of pending events before flushing to Kinesis.                                                                                                                                                                                                                                 |
| `collector.streams.buffer.timeLimit`                                      | Optional. Default: `5000`. Configures the maximum time in milliseconds before flushing pending buffered events to Kinesis.                                                                                                                                                                                                            |

### SQS collector options

| parameter                                                                 | description                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                                                  | Required. Name of the output SQS queue for successfully collected events.                                                                                                                                                                                                                                                             |
| `collector.streams.bad`                                                   | Required. Name of the output SQS queue for http requests which could not be written to the good stream. For example, if the event size exceeds the SQS limit of 256KB.                                                                                                                                                                |
| `collector.streams.useIpAddressAsPartitionKey` (deprecated since _3.5.0_) | Optional. Default: `false`. Whether to use the user's IP address as the Kinesis partition key. This is attached to the SQS message as an attribute, with the aim of using it if the events ultimately end up in Kinesis.                                                                                                              |
| `collector.streams.{good,bad}.region`                                     | Optional. Default: `eu-central-1`. AWS region of the SQS queues.                                                                                                                                                                                                                                                                      |
| `collector.streams.{good,bad}.threadPoolSize`                             | Optional. Default: `10`. Thread pool size used by the collector sink for asynchronous operations.                                                                                                                                                                                                                                     |
| `collector.streams.{good,bad}.aws.accessKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_ACCESS_KEY_ID` environment variable.                                                                                                                                                                     |
| `collector.streams.{good,bad}.aws.secretKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_SECRET_ACCESS_KEY` environment variable.                                                                                                                                                                 |
| `collector.streams.{good,bad}.maxBytes` (since _2.9.0_)                   | Optional. Default: `192000` (192 kb). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead. SQS has a record size limit of 256 kb, but records are encoded with Base64, which adds approximately 33% of the size, so we set the limit to `256 kb * 3/4`. |
| `collector.streams.{good,bad}.startupCheckInterval` (since _2.9.0_)       | Optional. Default: `1 second`. When collector starts, it checks if SQS buffers exist with `getQueueUrl`. This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted.                                                                                  |
| `collector.streams.backoffPolicy.minBackoff`                              | Optional. Default: `3000`. Time (in milliseconds) for retrying sending to SQS after failure.                                                                                                                                                                                                                                          |
| `collector.streams.backoffPolicy.maxBackoff`                              | Optional. Default: `600000`. Time (in milliseconds) for retrying sending to SQS after failure.                                                                                                                                                                                                                                        |
| `collector.streams.buffer.byteLimit`                                      | Optional. Default: `3145728`. Incoming events are stored in an internal buffer before being sent to SQS. This configures the maximum total size of pending events.                                                                                                                                                                    |
| `collector.streams.buffer.recordLimit`                                    | Optional. Default: `500`. Configures the maximum number of pending events before flushing to SQS.                                                                                                                                                                                                                                     |
| `collector.streams.buffer.timeLimit`                                      | Optional. Default: `5000`. Configures the maximum time in milliseconds before flushing pending buffered events to SQS.                                                                                                                                                                                                                |

### Pubsub collector options

| parameter                                                                                        | description                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                                                                         | Required. Name of the output Pubsub topic for successfully collected events.                                                                                                                                                                          |
| `collector.streams.bad`                                                                          | Required. Name of the output pubsub topic for http requests which could not be written to the good stream. For example, if the event size exceeds the Pubsub limit of 10MB.                                                                           |
| `collector.streams.sink.{good,bad}.googleProjectId`                                              | Required. GCP project name.                                                                                                                                                                                                                           |
| `collector.streams.sink.{good,bad}backoffPolicy.minBackoff` (deprecated since _3.5.0_)           | Optional. Default: `1000`. Time (in milliseconds) for retrying sending to Pubsub after failure.                                                                                                                                                       |
| `collector.streams.sink.{good,bad}.backoffPolicy.maxBackoff` (deprecated since _3.5.0_)          | Optional. Default: `1000`. Time (in milliseconds) for retrying sending to Pubsub after failure                                                                                                                                                        |
| `collector.streams.sink.{good,bad}.backoffPolicy.totalBackoff` (deprecated since _3.5.0_)        | Optional. Default: `9223372036854`. We set this to the maximum value so that we never give up on trying to send a message to pubsub.                                                                                                                  |
| `collector.streams.sink.{good,bad}.backoffPolicy.multipler` (deprecated since _3.5.0_)           | Optional. Default: `2`. Multiplier between two periods.                                                                                                                                                                                               |
| `collector.streams.sink.{good,bad}.backoffPolicy.initialRpcTimeout` (deprecated since _3.5.0_)   | Optional. Default: `10000`. Time (in milliseconds) before a RPC call to Pubsub is aborted and retried.                                                                                                                                                |
| `collector.streams.sink.{good,bad}.backoffPolicy.maxRpcTimeout` (deprecated since _3.5.0_)       | Optional. Default: `10000`. Maximum time (in milliseconds) before RPC call to Pubsub is aborted and retried.                                                                                                                                          |
| `collector.streams.sink.{good,bad}.backoffPolicy.rpcTimeoutMultipler` (deprecated since _3.5.0_) | Optional. Default: `2`. How RPC timeouts increase as they are retried.                                                                                                                                                                                |
| `collector.streams.sink.{good,bad}..maxBytes` (since _2.9.0_)                                    | Optional. Default: `10000000` (10 MB). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead.                                                                             |
| `collector.streams.sink.{good,bad}.startupCheckInterval` (since _2.9.0_)                         | Optional. Default: `1 second`. When collector starts, it checks if PubSub topics exist with `listTopics`. This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted. |
| `collector.streams.sink.{good,bad}.retryInterval` (since _2.9.0_)                                | Optional. Default: `10 seconds`. Collector uses built-in retry mechanism of PubSub API. In case of failure of these retries, the events are added to a buffer and every `retryInterval` collector retries to send them.                               |
| `collector.streams.{good,bad}.buffer.byteLimit`                                                  | Optional. Default: `1000000`. Incoming events are stored in an internal buffer before being sent to Pubsub. This configures the maximum total size of pending events                                                                                  |
| `collector.streams.{good,bad}.buffer.recordLimit`                                                | Optional. Default: `40`. Maximum number of pending events before flushing to Pubsub.                                                                                                                                                                  |
| `collector.streams.{good,bad}.buffer.timeLimit`                                                  | Optional. Default: `1000`. Maximum time (in milliseconds) before flushing pending buffered events to Pubsub.                                                                                                                                          |

### Kafka collector options

| parameter                                           | description                                                                                                                                                                                                                         |
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                            | Required. Name of the output Kafka topic for successfully collected events.                                                                                                                                                         |
| `collector.streams.bad`                             | Required. Name of the output Kafka topic for http requests which could not be written to the good stream.                                                                                                                           |
| `collector.streams.{good,bad}.brokers`              | Required. A list of host:port pairs to use for establishing the initial connection to the Kafka cluster.                                                                                                                            |
| `collector.streams.{good,bad}.producerConf`         | Optional. Kafka producer configuration. See [the docs](https://kafka.apache.org/documentation/#producerconfigs) for all properties.                                                                                                 |
| `collector.streams.{good,bad}.maxBytes`             | Optional. Default: `1000000` (1 MB). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead.                                                             |
| `collector.streams.{good,bad}.startupCheckInterval` | Optional. Default: `10 second`. When collector starts, it checks if Kafka topics exist. This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted. |
| `collector.streams.{good,bad}.buffer.byteLimit`     | Optional. Default: `3145728`. Incoming events are stored in an internal buffer before being sent to Kafka. This configures the maximum total size of pending events.                                                                |
| `collector.streams.{good,bad}.buffer.recordLimit`   | Optional. Default: `500`. Configures the maximum number of pending events before flushing to Kafka.                                                                                                                                 |
| `collector.streams.{good,bad}.buffer.timeLimit`     | Optional. Default: `5000`. Configures the maximum time in milliseconds before flushing pending buffered events to Kafka.                                                                                                            |
| `collector.streams.{good,bad}.retryInterval`        | Optional. Default: `10 seconds`. Collector uses built-in retry mechanism of Kafka API. In case of failure of these retries, the events are added to a buffer and every `retryInterval` collector retries to send them.              |

### Setting the domain name

Set the cookie name using the `collector.cookie.name` setting. To maintain backward compatibility with earlier versions of the collector, use the string "sp" as the cookie name.

The collector responds to valid requests with a `Set-Cookie` header, which may or may not specify a `domain` for the cookie.

If no domain is specified, the cookie will be set against the full collector domain, for example `collector.snplow.com`. That will mean that applications running elsewhere on `*.snplow.com` won't be able to access it. If you don't need to grant access to the cookie from other applications on the domain, then you can ignore the `domains` and `fallbackDomain` settings.

In earlier versions, you could specify a `domain` to tie the cookie to. For example, if set to `.snplow.com`, the cookie would have been accessible to other applications running on `*.snplow.com`. To do the same in this version, use the `fallbackDomain` setting but **make sure** that you no longer include a leading dot:

```properties
fallbackDomain = "snplow.com"
```

The cookie set by the collector can be treated differently by browsers, depending on whether it's considered to be a first-party or a third-party cookie. In earlier versions (0.15.0 and earlier), if you had two collector endpoints, one on `collector.snplow.com` and one on `collector.snplow.net`, you could only specify one of those domains in the configuration. That meant that you were only able to set a first-party cookie server-side on either `.snplow.com` or `.snplow.net`, but not on both. From version 0.16.0, you can specify a list of domains to be used for the cookie (**note the lack of a leading dot**):

```properties
domains = [

  "snplow.com"

  "snplow.net"

]
```

Which domain to be used in the `Set-Cookie` header is determined by matching the domains from the `Origin` header of the request to the specified list. The first match is used. If no matches are found, the fallback domain will be used, if configured. If no `fallbackDomain` is configured, the cookie will be tied to the full collector domain.

If you specify a main domain in the list, all subdomains on it will be matched. If you specify a subdomain, only that subdomain will be matched.

Examples:

- `domain.com` will match `Origin` headers like `domain.com`, `www.domain.com` and `secure.client.domain.com`
- `client.domain.com` will match an `Origin` header like `secure.client.domain.com` but not `domain.com` or `www.domain.com`.

### Configuring custom paths

The collector responds with a cookie to requests with a path that matches the `vendor/version` protocol. The expected values are:

- `com.snowplowanalytics.snowplow/tp2` for Tracker Protocol 2
- `r/tp2` for redirects
- `com.snowplowanalytics.iglu/v1` for the Iglu Webhook.

You can also map any valid (ie, two-segment) path to one of the three defaults via the `collector.paths` section of the configuration file. Your custom path must be the key and the value must be one of the corresponding default paths. Both must be full valid paths starting with a leading slash:

```properties
paths {

  "/com.acme/track"    = "/com.snowplowanalytics.snowplow/tp2"

  "/com.acme/redirect" = "/r/tp2"

  "/com.acme/iglu"     = "/com.snowplowanalytics.iglu/v1"

}
```

### TLS port binding and certificate (2.4.0+)

Since 2.4.0 TLS certificates are configured using JVM system parameters. The "`Customizing JSSE`" section in [Java 11 JSSE reference documentation](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-A41282C3-19A3-400A-A40F-86F4DA22ABA9) explains all system properties in detail.

The following JVM properties are the ones to be used most of the time.

| System Property                  | Customized Item                                                                                                                                                                                                                                                           | Default                                                                                                                                                                                                                        | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `javax.net.ssl.keyStore`         | Default keystore; see [Customizing the Default Keystores and Truststores, Store Types, and Store Passwords](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-7D9F43B8-AABF-4C5B-93E6-3AFB18B66150)          | None                                                                                                                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `javax.net.ssl.keyStorePassword` | Default keystore password; see [Customizing the Default Keystores and Truststores, Store Types, and Store Passwords](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-7D9F43B8-AABF-4C5B-93E6-3AFB18B66150) | None                                                                                                                                                                                                                           | It is inadvisable to specify the password in a way that exposes it to discovery by other users. **Password can not be empty.**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `javax.net.ssl.keyStoreType`     | Default keystore type; see [Customizing the Default Keystores and Truststores, Store Types, and Store Passwords](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-7D9F43B8-AABF-4C5B-93E6-3AFB18B66150)     | `PKCS12`                                                                                                                                                                                                                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `jdk.tls.server.cipherSuites`    | Server-side default enabled cipher suites. See [Specifying Default Enabled Cipher Suites](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-D61663E8-2405-4B2D-A1F1-B8C7EA2688DB)                            | See [SunJSSE Cipher Suites](https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2__SUNJSSE_CIPHER_SUITES) to determine which cipher suites are enabled by default | Caution: These system properties can be used to configure weak cipher suites, or the configured cipher suites may be weak in the future. It is not recommended that you use these system properties without understanding the risks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `jdk.tls.server.protocols`       | Default handshaking protocols for TLS/DTLS servers. See [The SunJSSE Provider](https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2)                                                                        | None                                                                                                                                                                                                                           | To configure the default enabled protocol suite in the server-side of a SunJSSE provider, specify the protocols in a comma-separated list within quotation marks. The protocols in this list are standard SSL protocol names as described in [Java Security Standard Algorithm Names](https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html). Note that this System Property impacts only the default protocol suite (SSLContext of the algorithms SSL, TLS and DTLS). If an application uses a version-specific SSLContext (SSLv3, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3, DTLSv1.0, or DTLSv1.2), or sets the enabled protocol version explicitly, this System Property has no impact. |

A deployment would need a TLS certificate, preferably issued by a trusted CA, however, for test purposes, a TLS cert could be generated as follows

```bash
ssl_dir=/opt/snowplow/ssl

mkdir -p ${ssl_dir}



sudo openssl req \

  -x509 \

  -newkey rsa:4096 \

  -keyout ${ssl_dir}/collector_key.pem \

  -out ${ssl_dir}/collector_cert.pem \

  -days 3650 \

  -nodes \

  -subj "/C=UK/O=Acme/OU=DevOps/CN=*.acme.com"



sudo openssl pkcs12 \

  -export \

  -out ${ssl_dir}/collector.p12 \

  -inkey ${ssl_dir}/collector_key.pem \

  -in ${ssl_dir}/collector_cert.pem \

  -passout pass:changeme



sudo chmod 644 ${ssl_dir}/collector.p12
```

and then the collector (kinesis as example) could be started as follows

```bash
config_dir=/opt/snowplow/config



docker run \

  -d \

  --name scala-stream-collector \

  --restart always \

  --network host \

  -v ${config_dir}:/snowplow/config \

  -v ${ssl_dir}:/snowplow/ssl \

  -p 8080:8080 \

  -p 8443:8443 \

  -e 'JAVA_OPTS=-Xms2g -Xmx2g -Djavax.net.ssl.keyStoreType=pkcs12 -Djavax.net.ssl.keyStorePassword=changeme -Djavax.net.ssl.keyStore=/snowplow/ssl/collector.p12 -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -Dcom.amazonaws.sdk.disableCbor' \

  snowplow/scala-stream-collector-kinesis:2.5.0 \

  --config /snowplow/config/snowplow-stream-collector-kinesis-2.5.0.hocon
```

Note: If you don't have a verified certificate, you need to disable SSL verification on client side. e.g. `cURL`'s `-k, --insecure` flag disables it.

### Setting up an SQS buffer (2.0.0+)

On AWS, the lack of auto-scaling in Kinesis results in throttled streams in case of traffic spikes and the collector starts accumulating events to retry them later. If accumulation continues long enough, the collector will run out of memory. To prevent the possibility of a broken collector, we decided to make it possible to configure an SQS buffer which can provide additional assurance during extreme traffic spikes.

SQS is used to queue any message that the collector failed to send to Kinesis. The [Snowbridge application](/docs/api-reference/snowbridge/) can then read the messages from SQS and write them to Kinesis once Kinesis is ready. In the event of any AWS API glitches, there is a retry mechanism which retries sending to the SQS queue 10 times.

The keys set up for the Kinesis stream are stored as SQS message attributes in order to preserve the information.

> **Note:** The SQS messages cannot be as big as Kinesis messages. The limit is 256kB per message, but we send the messages as Base64 encoded, so the limit goes down to 192kB for the original message.

#### Setting up the SQS queues

> **Note:** This section only applies to the case when SQS is used as a fallback sink when Kinesis is unavailable. If you are using SQS as the primary sink, then the settings below should be ignored and the `good` and `bad` streams should be configured as normal under `streams.good` and `streams.bad` respectively.

To start using this feature, you will first need to set up the SQS queues. Two separate queues are required for good (raw) events and bad events. The Collector then needs to be informed about the queue names, and this can be done by adding these as entries to `config.hocon`:

```properties
sqsGoodBuffer = {good-sqs-queue-url}

sqsBadBuffer = {bad-sqs-queue-url}
```

### Networking

Since version 3.0.0 networking settings are configured in its own `collector.networking` section:

| parameter                                                    | description                                                                                                                                                                                                      |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.networking.maxConnections` (since _3.0.0_)        | Optional. Default: `1024`. Maximum number of concurrent active connection.                                                                                                                                       |
| `collector.networking.idleTimeout` (since _3.0.0_)           | Optional. Default: `610 seconds`. Maximum inactivity time for a network connection. If no data is sent within that time, the connection is closed.                                                               |
| `collector.networking.bodyReadTimeout` (since _3.7.0_)       | Optional. Default: `25 seconds`. Maximum time from receiving request headers to receiving end of request body. If exceeded returns a 408 Request Timeout.                                                        |
| `collector.networking.responseHeaderTimeout` (since _3.2.0_) | Optional. Default: `30 seconds`. Time from when the request is made until a response line is generated before a 503 response is returned. It is recommended to make this slightly larger than `bodyReadTimeout`. |
| `collector.networking.maxRequestLineLength` (since _3.2.0_)  | Optional. Default: `20480`. Maximum request line to parse. If exceeded returns a 400 Bad Request.                                                                                                                |
| `collector.networking.maxHeadersLength` (since _3.2.0_)      | Optional. Default: `40960`. Maximum data that composes the headers. If exceeded returns a 400 Bad Request.                                                                                                       |
| `collector.networking.maxPayloadSize` (since _3.3.0_)        | Optional. Default: `1048576` (1 MB). Maximum size of an event within payload allowed before emitting an Size Violation event. Returns 200 OK.                                                                    |
| `collector.networking.dropPayloadSize` (since _3.3.0_)       | Optional. Default: `2097152` (2 MB). Maximum body payload size allowed before rejecting the request. If exceeded returns a 413 Payload Too Large.                                                                |

---

# Introduction to Snowplow Collector
> The Snowplow event Collector receives raw Snowplow events from trackers and webhooks, serializes them, and writes them to supported sinks including Kinesis, PubSub, Kafka, NSQ, SQS, and stdout.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/

The collector receives raw Snowplow events sent over HTTP by [trackers](/docs/sources/) or [webhooks](/docs/sources/webhooks/). It serializes them, and then writes them to a sink. Currently supported sinks are:

1. [Amazon Kinesis](http://aws.amazon.com/kinesis/)
2. [Google PubSub](https://cloud.google.com/pubsub/)
3. [Apache Kafka](http://kafka.apache.org/)
4. [NSQ](http://nsq.io/)
5. [Amazon SQS](https://aws.amazon.com/sqs/)
6. `stdout` for a custom stream collection process

The collector supports cross-domain Snowplow deployments, setting a `user_id` (used to identify unique visitors) server side to reliably identify the same user across domains.

## How it works

### User identification

The collector allows the use of a third-party cookie, making user tracking across domains possible.

In a nutshell: the collector receives events from a tracker, sets/updates a third-party user tracking cookie, and returns the pixel to the client. The ID in this third-party user tracking cookie is stored in the `network_userid` field in Snowplow events.

In pseudocode terms:

```text
if (request contains an "sp" cookie) {

    Record that cookie as the user identifier

    Set that cookie with a now+1 year cookie expiry

    Add the headers and payload to the output array

} else {

    Set the "sp" cookie with a now+1 year cookie expiry

    Add the headers and payload to the output array

}
```

## Technical architecture

The collector is written in scala and built on top of [http4s](https://http4s.org).

[GitHub repository](https://github.com/snowplow/stream-collector)

---

# Set up and run Collector
> Instructions for running the Snowplow event Collector using Docker images or jar files, including how to configure the application with HOCON files and perform health checks.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/setup/

A Terraform module is available which deploys the collector on a AWS EC2 without the need for this manual setup.

## Run the collector

The collector is on docker hub with several different flavours. Pull the image that matches the sink you are using:

```bash
docker pull snowplow/scala-stream-collector-kinesis:3.7.0

docker pull snowplow/scala-stream-collector-pubsub:3.7.0

docker pull snowplow/scala-stream-collector-kafka:3.7.0

docker pull snowplow/scala-stream-collector-rabbitmq-experimental:3.7.0

docker pull snowplow/scala-stream-collector-nsq:3.7.0

docker pull snowplow/scala-stream-collector-sqs:3.7.0

docker pull snowplow/scala-stream-collector-stdout:3.7.0
```

The application is configured by passing a hocon file on the command line:

```bash
docker run --rm \

-v $PWD/config.hocon:/snowplow/config.hocon \

-p 8080:8080 \

snowplow/scala-stream-collector-${flavour}:3.7.0 --config /snowplow/config.hocon
```

Alternatively, you can download and run [a jar file from the github release](https://github.com/snowplow/stream-collector/releases).

```bash
java -jar scala-stream-collector-kinesis-3.7.0.jar --config /path/to/config.hocon
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Collector (since version 2.4.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `collector.telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `collector.telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

## Health check

Pinging the collector on the /health path should return a 200 OK response:

```bash
curl http://localhost:8080/health
```

---

# Collector 3.0.x upgrade guide
> Upgrade guide for Collector 3.0.x covering breaking changes including the new license, migration from Akka HTTP to http4s, single endpoint port, and updated configuration requirements.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/upgrade-guides/3-0-x-upgrade-guide/

## Breaking changes

### New license

Since version 3.0.0, the collector has been migrated to use the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

### New HTTP stack

Version 3.0.0 replaces the Akka HTTP stack with http4s. The same stack is already used in all other Snowplow microservices, so this makes the codebase more uniform and enables us to share more code between applications.

Also, newer versions of Akka HTTP would have required a [commercial license from Lightbend](https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka) for many of Snowplow’s users.

### Single endpoint port

Previously, the collector was able to expose both http and https ports. We’ve found it impractical and therefore the new version is exposing a single port — either http or https. To enable http→https upgrades, use a load-balancer/proxy.

## Upgrading

In order to ease the migration for existing installations, we’ve tried to keep the configurations mostly backwards-compatible.

However, due to changes above, some amendments are required. We’ve also introduced a few tweaks that should simplify configurations between pipeline services. Full configuration examples are available in [snowplow/stream-collector](https://github.com/snowplow/stream-collector/tree/master/examples) repository along with extended commentary.

If you’ve previously used both HTTP and HTTPS ports in the collector, this feature is no longer available. Since version 3.0.0 you can either:

- enable TLS termination in the collector (use HTTPS between load balancer and collector) and enable HTTP upgrade in load balancer running in front of the collector
- disable TLS termination in the collector (use HTTP between load balancer and collector), use TLS termination and HTTP upgrade in the load balancer

### License acceptance

You have to explicitly accept the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)). To do so, either set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable, or update the following section in the configuration:

```hcl
collector {

  license {

    accept = true

  }

  ...

}
```

### Akka section removal

The top-level `akka` section will no longer be used. Keeping the section in your configuration will _not_ cause collector failures but should be removed for clarity.

### Split sinks

Sink configurations can now be defined similarly as with our other services. Each sink is configured separately to allow for individual config specifics. This change is especially useful for our Kafka collector running on Azure, where EventHubs needs separate `producerConf` sections to account for different EH settings.

Example:

```hcl
collector {

  ...

  streams {

    good {

      name = "good"

      brokers = "localhost:9092,another.host:9092"

      producerConf {

        acks = all

        "key.serializer"     = "org.apache.kafka.common.serialization.StringSerializer"

        "value.serializer"   = "org.apache.kafka.common.serialization.StringSerializer"

      }

      buffer {

        byteLimit = 3145728

        recordLimit = 500

        timeLimit = 5000

      }

    }



    bad {

      name = "bad"

      brokers = "localhost:9092,another.host:9092

      producerConf {

        acks = all

        "key.serializer"     = "org.apache.kafka.common.serialization.StringSerializer"

        "value.serializer"   = "org.apache.kafka.common.serialization.StringSerializer"

      }

      maxBytes = 1000000

      buffer {

        byteLimit = 3145728

        recordLimit = 500

        timeLimit = 5000

      }

    }

  }

  ...

}
```

### Networking

Optional networking settings were previously a part of Akka HTTP configuration section. With the removal of the framework, the relevant settings have been moved into a dedicated `collector.networking` section:

- `collector.networking.maxConnections` - maximum number of concurrent active connection.
- `collector.networking.idleTimeout` - maximum inactivity time for a network connection. If no data is sent within that time, the connection is closed.

Example:

```hcl
collector {

  ...

  networking {

    maxConnections = 1024

    idleTimeout = 610 seconds

  }

  ...

}
```

---

# Collector 3.6.x upgrade guide with compression
> Upgrade guide for Collector 3.6.x introducing payload compression to reduce storage costs and improve throughput, requiring Enrich 6.1.x for compatibility.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/upgrade-guides/3-6-x-upgrade-guide/

Collector 3.6.0 introduces payload compression, a new feature that significantly reduces the size (and therefore, cost) of data written to your raw output stream.

The compression feature allows the collector to batch multiple individual collector payloads into a single compressed stream record. This provides several benefits:

- **Reduced storage costs**: compressed payloads take up less space in your output streams
- **Improved throughput**: fewer, larger records reduce the overhead of stream processing
- **Better performance**: downstream consumers can process batches more efficiently

## Enabling compression

> **Danger:** Before enabling compression, you must upgrade to Enrich 6.1.x first. The reason for this is that support for processing compressed payloads is added to Enrich starting with Enrich 6.1.0. Enrich 6.1.0 can process both compressed and uncompressed payloads.
>
> Enrich is currently the only application compatible with compression. Setups with an [S3 loader](/docs/api-reference/loaders-storage-targets/s3-loader/) reading off the raw stream will not be supported.

When upgrading to Collector 3.6.0, compression is an optional feature that can be configured in your [collector settings](/docs/api-reference/stream-collector/configure/). If this feature is not enabled, there will be no changes to the data format or size.

After upgrading Enrich, compression in Collector can be enabled by adding the following config section:

```hocon
compression {

  enabled = true

}
```

> **Tip:** Take note of the `streams.buffer.timeLimit` Collector configuration parameter, which already existed in previous versions. This controls how many events are batched (or, technically, for how long) before appling compression. Bigger values lead to higher compression rates (lower infrastructure costs), but also higher latency. We recommend starting with a value around 300–500ms and fine-tuning it from there.

### Impact on metrics

When compression is enabled, there will be a big decrease in the number of messages sent to the `raw` event stream, i.e. Kinesis, Pub/Sub or Event Hubs, depending on your cloud. You will notice this decrease if you monitor metrics on messages in the `raw` stream.

This is perfectly normal and does not indicate any drop in event volumes. It happens because the compression feature batches together many Snowplow events into a single message sent to the `raw` stream.

---

# Collector upgrade guides
> Guides to help you upgrade the Snowplow event Collector to newer versions with information on breaking changes and migration steps.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/upgrade-guides/

This section contains information to help you upgrade to newer versions of Collector.

## [3.6.x upgrade guide](/docs/api-reference/stream-collector/upgrade-guides/3-6-x-upgrade-guide/)

[Upgrade guide for Collector 3.6.x introducing payload compression to reduce storage costs and improve throughput, requiring Enrich 6.1.x for compatibility.](/docs/api-reference/stream-collector/upgrade-guides/3-6-x-upgrade-guide/)

## [3.0.x upgrade guide](/docs/api-reference/stream-collector/upgrade-guides/3-0-x-upgrade-guide/)

[Upgrade guide for Collector 3.0.x covering breaking changes including the new license, migration from Akka HTTP to http4s, single endpoint port, and updated configuration requirements.](/docs/api-reference/stream-collector/upgrade-guides/3-0-x-upgrade-guide/)

---

# Links to tracker API documentation
> Generated API documentation for Snowplow tracker SDKs across all supported platforms and programming languages.
> Source: https://docs.snowplow.io/docs/api-reference/trackers/

This section contains links to the generated API documentation for our tracker SDKs.

---

# Snowplow component versions and compatibility matrix
> Latest versions of Snowplow components including collectors, enrichment, loaders, trackers, Iglu, data models, and analytics SDKs with compatibility and upgrade information.
> Source: https://docs.snowplow.io/docs/api-reference/versions/

This page lists the most recent versions of Snowplow components.

Some information about components is relevant only for [Snowplow Self-Hosted](/docs/get-started/#self-hosted) users, as [Snowplow CDI](/docs/get-started/#customer-data-infrastructure) customers won't need to configure all their own components.

In short, almost everything is compatible with almost everything. We rarely change the core protocols that various components use to communicate.

You might encounter specific restrictions when following the documentation, for example, some of our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) might call for a reasonably recent version of the [warehouse loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/).

## Upgrades and deprecation

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, you don't need to deal with upgrading your pipeline. We'll perform upgrades for you.

Some major upgrades might have breaking changes. In this case, we provide upgrade guides, such as the ones for [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/).

From time to time, we develop better applications for certain tasks and deprecate the old ones. Deprecations are announced on [Community](https://community.snowplow.io/).

***

## Latest versions

### Core pipeline

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, you don't need to install any of the core pipeline components yourself. We'll deploy your pipeline and keep it up to date.

**AWS:**

| Component                                                                                                        | Latest version |
| ---------------------------------------------------------------------------------------------------------------- | -------------- |
| [Stream Collector](/docs/api-reference/stream-collector/)                                                        | 3.7.0          |
| [Enrich](/docs/api-reference/enrichment-components/)                                                             | 6.11.0         |
| [RDB Loader (Redshift, Snowflake, Databricks)](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) | 6.6.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                          | 0.11.0         |
| [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)            | 0.6.1          |
| [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/)          | 0.5.0          |
| [S3 Loader](/docs/api-reference/loaders-storage-targets/s3-loader/)                                              | 3.2.0          |
| [Snowbridge](/docs/api-reference/snowbridge/)                                                                    | 5.2.0          |
| [Elasticsearch Loader](/docs/api-reference/loaders-storage-targets/elasticsearch/)                               | 3.0.1          |
| [Postgres Loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/)                         | 0.3.3          |
| [Dataflow Runner](/docs/api-reference/dataflow-runner/)                                                          | 0.7.8          |

**GCP:**

| Component                                                                                               | Latest version |
| ------------------------------------------------------------------------------------------------------- | -------------- |
| [Stream Collector](/docs/api-reference/stream-collector/)                                               | 3.7.0          |
| [Enrich](/docs/api-reference/enrichment-components/)                                                    | 6.11.0         |
| [RDB Loader (Snowflake, Databricks)](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)  | 6.6.0          |
| [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/)                         | 2.2.1          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                 | 0.11.0         |
| [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)   | 0.6.1          |
| [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) | 0.5.0          |
| [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/)                  | 0.5.7          |
| [Snowbridge](/docs/api-reference/snowbridge/)                                                           | 5.2.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                 | 0.11.0         |
| [Postgres Loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/)                | 0.3.3          |

**Azure:**

| Component                                                                                               | Latest version |
| ------------------------------------------------------------------------------------------------------- | -------------- |
| [Stream Collector](/docs/api-reference/stream-collector/)                                               | 3.7.0          |
| [Enrich](/docs/api-reference/enrichment-components/)                                                    | 6.11.0         |
| [RDB Loader (Snowflake)](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)              | 6.6.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                 | 0.11.0         |
| [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)   | 0.6.1          |
| [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) | 0.5.0          |

***

### Iglu (schema registry)

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, you don't need to install Iglu Server yourself. It's also unlikely that you need to use any of the other components in this section. You can manage your data structures [in the UI or via the API](/docs/event-studio/data-structures/).

| Component                                                              | Latest version |
| ---------------------------------------------------------------------- | -------------- |
| [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) | 0.14.1         |
| [`igluctl` utility](/docs/api-reference/iglu/igluctl/)                 | 0.13.0         |

### Trackers

| Tracker                                                     | Latest version |
| ----------------------------------------------------------- | -------------- |
| [JavaScript (Web and Node.js)](/docs/sources/web-trackers/) | 4.8.0          |
| [iOS](/docs/sources/mobile-trackers/)                       | 6.2.1          |
| [Android](/docs/sources/mobile-trackers/)                   | 6.2.0          |
| [React Native](/docs/sources/react-native-tracker/)         | 4.8.0          |
| [Flutter](/docs/sources/flutter-tracker/)                   | 0.8.0          |
| [WebView](/docs/sources/webview-tracker/)                   | 0.3.0          |
| [Roku](/docs/sources/roku-tracker/)                         | 0.3.1          |
| [Google AMP](/docs/sources/google-amp-tracker/)             | 1.1.0          |
| [Pixel](/docs/sources/pixel-tracker/)                       | 0.3.0          |
| [Golang](/docs/sources/golang-tracker/)                     | 3.1.0          |
| [.NET](/docs/sources/net-tracker/)                          | 1.3.0          |
| [Java](/docs/sources/java-tracker/)                         | 2.1.0          |
| [Python](/docs/sources/python-tracker/)                     | 1.0.3          |
| [Scala](/docs/sources/scala-tracker/)                       | 2.0.0          |
| [Ruby](/docs/sources/ruby-tracker/)                         | 0.8.0          |
| [Rust](/docs/sources/rust-tracker/)                         | 0.2.0          |
| [PHP](/docs/sources/php-tracker/)                           | 0.9.2          |
| [C++](/docs/sources/c-tracker/)                             | 2.0.0          |
| [Unity](/docs/sources/unity-tracker/)                       | 0.8.1          |
| [Lua](/docs/sources/lua-tracker/)                           | 0.2.0          |

### Data models

#### dbt

[Modeling data with dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/) is our recommended approach.

**Snowplow Unified Digital:**

| snowplow-unified version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ------------------------ | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                    | >=1.10.6 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.4.5                    | >=1.6.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |

**Snowplow Media Player:**

| snowplow-media-player version | snowplow-web version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ----------------------------- | -------------------- | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.1                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.8.0                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.5.3                         | >=0.14.0 to <0.16.0  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.2                         | >=0.13.0 to <0.14.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.1                         | >=0.12.0 to <0.13.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.3.4                         | >=0.9.0 to <0.12.0   | >=1.0.0 to <1.3.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.1.0                         | >=0.6.0 to <0.7.0    | >=0.20.0 to <1.1.0 | ❌        | ❌          | ✅        | ❌         | ✅        | ❌     |

**Snowplow Normalize:**

| snowplow-normalize version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ✅     |
| 0.3.5                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.3                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.1.0                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

**Snowplow Ecommerce:**

| snowplow-ecommerce version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.1                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ✅     |
| 0.8.2                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ❌     |
| 0.3.0                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.1                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

Postgres is technically supported in the models within the package, however one of the contexts’ names is too long to be loaded via the Postgres Loader.

**Snowplow Attribution:**

| snowplow-attribution version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ---------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ✅     |
| 0.3.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ❌     |

***

See also the [dbt version compatibility checker](/docs/modeling-your-data/modeling-your-data-with-dbt/#dbt-version-compatibility-checker).

#### SQL Runner

> **Note:** We recommend using the dbt models above, as they are more actively developed.

The latest version of [SQL Runner](/docs/modeling-your-data/modeling-your-data-with-sql-runner/) itself is **0.10.1**.

| Model                                                                                               | Redshift | BigQuery | Snowflake |
| --------------------------------------------------------------------------------------------------- | -------- | -------- | --------- |
| [Web](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/)       | 1.3.1    | 1.0.4    | 1.0.2     |
| [Mobile](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-mobile-data-model/) | 1.1.0    | 1.1.0    | 1.1.0     |

### Testing and debugging

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, we recommend using [Snowplow Micro through Console](/docs/testing/snowplow-micro/console/) for testing and debugging.

| Application                                                     | Latest version |
| --------------------------------------------------------------- | -------------- |
| [Snowplow Micro](/docs/testing/snowplow-micro/)                 | 4.4.0          |
| [Snowplow Mini](/docs/api-reference/snowplow-mini/usage-guide/) | 0.26.0         |

### Analytics SDKs

| SDK                                                                       | Latest version |
| ------------------------------------------------------------------------- | -------------- |
| [Scala](/docs/api-reference/analytics-sdk/analytics-sdk-scala/)           | 3.0.0          |
| [Javascript](/docs/api-reference/analytics-sdk/analytics-sdk-javascript/) | 0.3.1          |
| [Python](/docs/api-reference/analytics-sdk/analytics-sdk-python/)         | 0.2.3          |
| [.NET](/docs/api-reference/analytics-sdk/analytics-sdk-net/)              | 0.2.1          |
| [Go](/docs/api-reference/analytics-sdk/analytics-sdk-go/)                 | 0.3.0          |

---

# Create event forwarders in Console
> Step-by-step guide to create a Snowplow event forwarder to send events to third-party destinations in real-time with connections, filters, and field mappings.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/creating-forwarders/

Follow the steps below to configure a new event forwarder. See [available integrations](/docs/destinations/forwarding-events/integrations/) for destination-specific guides.

## Step 1: Create a connection

A **connection** is a resource that stores the credentials and endpoint details needed to send events to your destination.

To create a connection from [Snowplow Console](https://console.snowplowanalytics.com), first go to **Settings** > **Connections**, then select **Set up connection**. From the dropdown, choose **Event forwarding**, then select the destination you want to forward events to. Each destination will have specific authentication and endpoint details required.

![Console interface for creating a new destination connection with authentication and endpoint configuration fields](/assets/images/event-forwarding-connection-094b95e8dd4dee4353cd575fab0468b6.png)

When finished, click **Deploy**. Once a connection is deployed, you can use it in one or more forwarders to connect to your destination.

## Step 2: Create a new forwarder

1. Go to **Destinations** > **Event forwarding**.
2. Click **Add destination** and select the destination from the list of available integrations to start setting up the forwarder.
3. Give the forwarder a **name**, select the **pipeline** you want the forwarder to read events from, and choose the **connection** you created in step 1.
4. Optionally, you can choose to **Import configuration from** an existing forwarder. This is helpful when migrating a forwarder setup from development to production.
5. Click **Continue** to configure event filters and data mapping.

## Step 3: Configure event filters and field mapping

Forwarders use JavaScript expressions to define which events to forward and how to map Snowplow data to your destination's required schema.

### Event filtering

Use JavaScript expressions to select which events to forward. Only events that return `true` when evaluated against your filter will be sent to your destination.

Use the `event` object to reference fields on your Snowplow payloads. For example:

```javascript
// Forward page views from website

event.app_id == "website" && event.event_name == "page_view"



// Forward a list of custom events

["add_to_cart", "purchase"].includes(event.event_name)
```

Leave the filter blank to forward all events.

![Event filtering configuration panel showing JavaScript expression field for defining which events to forward](/assets/images/event-forwarding-filters-eb59897842efc2acb3dd85500b569aae.png)

### Field mapping

Define how Snowplow data maps to your destination fields. For each mapping, **Destination Field** represents the property name and **Snowplow expression** is a JavaScript expression used to extract data from your Snowplow event. Snowplow provides default mappings based on common fields, but you can overwrite or delete them as needed.

![Field mapping configuration panel showing key/value pairs with JavaScript property selection expressions](/assets/images/event-forwarding-mapping-0811c90079f2c0d53d555f11294bad1a.png)

### Custom functions

You can also write JavaScript functions for complex data transformations. Examples include converting date formats, transforming enum values, combining multiple fields, or applying other business logic. You can then reference functions in both the event filter and field mapping sections.

![Custom functions editor with JavaScript code for complex data transformations in event forwarding](/assets/images/event-forwarding-custom-functions-7a4098875609b9bcb07169204c459fbc.png)

> **Info:** To learn more about the supported filter and mapping expressions, check out the [filter and mapping reference](/docs/destinations/forwarding-events/reference/).

## Step 4: Test your transformations

Once you've defined your filter and mapping configuration, you can test it against a sample event and preview what the output JSON payload looks like. Choose an event from the **Select sample input event** dropdown and selecting **Run test**.

![Test transformation interface showing sample event input and JSON output preview with run test button](/assets/images/event-forwarding-test-transformations-54e02b5e0465eca443b8f05774912721.png)

Snowplow provides a few out-of-the-box sample events to test with, which you can edit as needed. You can also choose **Custom event** to paste in your own JSON-formatted Snowplow event. You can use [Snowplow Micro](/docs/testing/snowplow-micro/) to collect some test events and copy their JSON payload for testing.

Select **View generated code** to see the JavaScript function generated from your filters, field mappings, and custom functions. This is exactly what will run when transforming events for your destination, and can be used directly in a [Snowbridge JavaScript transformation](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) for local testing.

If there is an error with your configuration, the generated code will show the line number which contains the error.

## Step 5: Deploy

When you're done, select **Deploy** to save your configuration and create the forwarder. This will deploy the underlying Snowbridge instance to your cloud account and begin forwarding events based on your configuration. It typically takes a few minutes to deploy a new forwarder.

---

# Build custom integrations for event streams
> Build custom consumers for Snowplow event streams using AWS Lambda, GCP Cloud Functions, KCL applications, or Pub/Sub client libraries to integrate with any third-party platform.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/custom-integrations/

Snowplow is underpinned by event streams: AWS Kinesis, GCP PubSub, or Apache Kafka. Before a Snowplow pipeline loads the events to a data warehouse, the enriched events are available on a stream. You can build a custom consumer to consume these events. Below we describe some high level concepts which can be used to consume the enriched event streams.

## Transforming the Enriched Stream to JSON

The Snowplow events in the Enriched stream are in a tab separated format (TSV) by default. Many downstream consumers will prefer this data in JSON format, and the [Snowplow Analytics SDKs](/docs/api-reference/analytics-sdk/) have been built to help with this.

## AWS Lambda and GCP Cloud Functions

[AWS Lambdas](https://aws.amazon.com/lambda/) and [GCP Cloud Functions](https://cloud.google.com/functions/) are server-less platforms that allow you to write applications that can be triggered by events from Kinesis and PubSub respectively. By configuring a function to be triggered by an event, it is possible to write applications that take the Snowplow events, perform transformations and other processing, then relay that event into another system.

Server-less functions are an easy way to approach building real time consumers of the event stream for those use cases which require fast action or decisioning based on incoming events (For example, Ad Bidding, Paywall Optimization, Real-time reporting, etc).

## Kinesis Client Library (KCL) applications

The KCL (Kinesis Consumer Library) allows for applications to be built to consume from AWS Kinesis. It makes use of AWS DynamoDB to keep track of shards in the data stream, and makes it far easier to consume from Kinesis than would otherwise be possible.

There is comprehensive documentation on building Amazon KCL apps within the [AWS Documentation](https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html).

## Pub/Sub client library applications

The Pub/Sub client libraries allow for applications to be built to consume from GCP Pub/Sub. It makes it easy to build against and consume events in Pub/Sub streams, ultimately making it far easier to consume from Pub/Sub than would otherwise be possible.

There is comprehensive documentation on building GCP Pub/Sub client library apps within the [GCP Documentation](https://cloud.google.com/pubsub/docs/reference/libraries).

---

# Monitor and troubleshoot event forwarders
> Monitor event forwarder performance, debug failures, and understand retry logic with cloud metrics, failed event logs, and Console statistics.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/event-forwarding-monitoring-and-troubleshooting/

This page outlines how to monitor event forwarder performance and diagnose delivery issues. Snowplow provides both summary metrics and detailed failed event logs to help you understand failure patterns and troubleshoot specific problems.

## Failure types and handling

Snowplow handles event forwarding failures differently depending on the type of error.

### Data processing failures

These failures occur when there are issues with the event data itself or how in how it's transformed before reaching the destination.

**Transformation failures** occur when Snowplow hits an exception when applying your configured JavaScript transformation. While there are safeguards against deploying invalid JavaScript, transformations may still result in runtime errors. Snowplow treats transformation failures as invalid data and logs them as failed events in your cloud storage bucket without retrying.

**Oversized data failures** results from events exceeding the destination's size limits. Snowplow creates [size violation failed events](/docs/api-reference/failed-events/) for these events and logs them to your cloud storage bucket without retrying.

### Destination failures

These failures occur when the destination's API cannot accept or process the event data.

**Transient failures** are those that are expected to succeed on retry. This includes temporary network errors, HTTP 5xx server errors, or rate limiting. Transient failures are automatically retried.

**Setup failures** result from configuration issues that typically require human intervention to resolve, such as invalid API keys or insufficient permissions. When a setup error occurs, Snowplow will trigger email alerts to the configured list of users and occassionally retry the request to check if the issue has been resolved. For more information on alerting, see [configuring setup alerts](#configuring-setup-alerts) on this page.

**Other unrecoverable failures** are bad requests that won't succeed on retry, such as those with missing or invalid fields. These often map to HTTP 400 response codes. Snowplow will log them as failed events in your cloud storage bucket without retrying.

Failure types are defined per destination based on their expected HTTP response codes. See the list of [available destinations](/docs/destinations/forwarding-events/integrations/) for destination-specific details on retry policies and error handling.

### What happens when events fail

When any type of failure occurs, Snowplow can take one or more of the following actions:

- **Automatic retries**: transient failures are automatically retried according to each destination's retry policy. For all HTTP API destinations, Snowplow will retry up to 5 times with exponential backoff.
- **Failed event logging**: all non-retryable failures are routed to your configured failure destination, which is typically a cloud storage bucket, where you can inspect them further. This includes transformation failures, oversized data failures, unrecoverable failures, and transient failures that have exceeded their retry limit. For how to query these logs, see [Inspecting and debugging failures](#inspecting-and-debugging-failures) on this page.
- **Setup alerts**: just like warehouse loaders, setup failures trigger email alerts to notify configured users of authentication or configuration problems.

## Configuring setup alerts

Once a forwarder is deployed, you can configure one or more email addresses to send alerts when setup failures occur. Follow the steps below to configure the alerts.

1. Navigate to **Destinations** > **Event forwarding** from the sidebar.
2. In the table of forwarders, click the three dots next to the forwarder you want to configure alerting for and select **Alerts**.
3. You'll see a modal where you can enter the email addresses you want to be alerted in case of setup errors. Click **Save Changes** to confirm.

![Setup alerts modal dialog for configuring email addresses to receive forwarder error notifications](/assets/images/setup-alerting-screenshot-a84253dd9889a28c38179557139d9425.png)

## Metrics and monitoring

You can monitor forwarders in a few ways:

- **Console metrics**: you can view high-level delivery statistics in Console.
- **Cloud monitoring metrics**: forwarders emit a set of metrics to your cloud provider's observability service.
- **Failed event logs**: for failed deliveries, Snowplow saves detailed logs to your cloud storage bucket. Consume these logs for automated monitoring in your observability platform of choice.

### Console metrics

In Snowplow Console, you can see the number of filtered, failed, and successfully delivered events over the last seven days.

To view these metrics, navigate to **Destinations** > **Event forwarding**. In the event forwarders table, you will see metrics for each forwarder.

![Console metrics dashboard showing forwarder statistics including filtered, failed, and delivered event counts](/assets/images/event-forwarding-console-metrics-a7c681591dad5e6ed07e55980b809c1c.png)

### Cloud monitoring metrics

> **Info:** Forwarder cloud metrics are only available for [CDI Private Managed Cloud](/docs/get-started/#cdi-private-managed-cloud) customers.

Forwarders emit the following metrics in your cloud provider's monitoring service:

- `target_success`: events successfully delivered to your destination
- `target_failed`: events that failed delivery but are eligible for retry
- `message_filtered`: events filtered out based on the forwarder's JavaScript filter expression
- `failure_target_success`: events that failed with unrecoverable errors, such as transformation errors, and logged to your cloud storage bucket

You can find forwarder metrics in the following locations based on which cloud provider you use:

- **AWS**: CloudWatch metrics under `snowplow/event-forwarding` namespace
- **GCP**: Cloud Monitoring metrics with `snowplow_event_forwarding` prefix

To get notified of any issues, you can use these metrics to define [CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) or [Cloud Monitoring alerts](https://cloud.google.com/monitoring/alerts).

## Inspecting and debugging failures

This section explains how to find and query failed event logs.

### Finding failed event logs

To better understand why a failure has occurred, you can directly access and review detailed failed delivery logs in file storage. The logs are automatically saved as [failed events](/docs/monitoring/exploring-failed-events/file-storage/) in your Snowplow cloud storage bucket under the prefix: `/{pipeline_name}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_errors/`

For more details on where to find failed events, see [Accessing failed events in file storage](/docs/monitoring/exploring-failed-events/file-storage/).

Failed event logs are formatted according to the [`event_forwarding_error`](https://iglucentral.com/?q=event_forwarding_error) schema and contain:

- **Original event data**: the complete Snowplow event that failed
- **Error details**: specific error type and message
- **Failure timestamp**: when the error occurred
- **Transformation state**: data state at the point of failure

### Querying failed event logs

You can query failed events using [Athena](https://aws.amazon.com/athena/) on AWS or [BigQuery external tables](https://cloud.google.com/bigquery/docs/external-tables) on GCP.

**AWS:**

**1. Create a table and load the data**

To make the logs easier to query, run the following query to create a table:

```sql
CREATE EXTERNAL TABLE snowplow_event_forwarding_failures (

  data struct<payload:string,

              processor:struct<artifact:string,

                              version:string

              >,

              failure:struct<errorCode:string,

                            errorMessage:string,

                            errorType:string,

                            latestState:string,

                            timestamp:string>

              >

  )

ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'

LOCATION 's3://{BUCKET_NAME}/{PIPELINE_NAME}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/'
```

If the table already exists, run the following query to pull in new data:

```sql
MSCK REPAIR TABLE event_forwarding_failures
```

**2. Explore failure records**

Use the query below to view a sample of failure records:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState, -- these are last because they can be quite large

    data.payload

FROM event_forwarding_failures

LIMIT 10
```

**3. Example queries**

Summarize the most common types of errors:

```sql
SELECT

    data.failure.errorType,

    data.failure.errorCode,

        -- time range for each - is the issue still happening?

    MIN(data.failure.timestamp) AS minTstamp,

    MAX(data.failure.timestamp) AS maxTstamp,

        -- How many errors overall,

    count(*) AS errorCount,

        -- There might just be lots of different messages for the same error

        -- If this close to error count, the messages for a single error might just have high cardinality - worth checking the messages themselves

        -- If it's a low number, we might have more than one issue

        -- If it's 1, we have only one issue and the below message is shared by all

    count(DISTINCT data.failure.errorMessage) AS distinctErrorMessages,

        -- a sample of error message. You may need to look at them individually to get the full picture

    MIN(data.failure.errorMessage) AS sampleErrorMessage

FROM event_forwarding_failures

GROUP BY 1, 2

ORDER BY errorCount DESC -- Most errors first

LIMIT 10
```

View transformation errors:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState,

    data.payload

FROM event_forwarding_failures

WHERE data.failure.errorType = 'transformation'

LIMIT 50
```

View API errors:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState,

FROM event_forwarding_failures

WHERE data.failure.errorType = 'api'

LIMIT 50
```

Filter based on a date and hour:

```sql
-- Note that the times in the paths are for the creation of the file, not the failure time



SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState, -- these are last because they can be quite large

    data.payload

FROM event_forwarding_failures

WHERE "$path" LIKE '%2025-07-29-16%' -- File paths are timestamped like this, so we can limit our queries this way

LIMIT 50
```

Filter for a range of timestamps:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState, -- these are last because they can be quite large

    data.payload

FROM event_forwarding_failures

-- Here we need the full path prefix

WHERE "$path" > 's3://{BUCKET_NAME}/{PIPELINE_NAME}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/2025-07-29-16'

AND "$path" < 's3://{BUCKET_NAME}/{PIPELINE_NAME}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/2025-07-29-20'
```

**GCP:**

**1. Create a dataset**

First, create a dataset to organize your failed event tables:

```sql
CREATE SCHEMA snowplow_failed_events

OPTIONS (

  description = "Dataset for Snowplow failed event analysis",

  location = "EU"  -- Should match the location of your bad rows bucket

);
```

**2. Create an external table and load the data**

To make the logs easier to query, run the following query to create an external table:

```sql
  CREATE OR REPLACE EXTERNAL TABLE snowplow_failed_events.event_forwarding_failures (

    schema STRING,

    data STRUCT<

      failure STRUCT<

        errorCode STRING,

        errorMessage STRING,

        errorType STRING,

        latestState STRING,

        timestamp TIMESTAMP

      >,

      payload STRING,

      processor STRUCT<

        artifact STRING,

        version STRING

      >

    >

  )

  OPTIONS (

    description="Event Forwarding failures",

    format="NEWLINE_DELIMITED_JSON",

    ignore_unknown_values=true,

    uris=["gs://{BUCKET}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/*"]

  );
```

**3. Explore failure records**

Use the query below to view a sample of failure records:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState, -- these are last because they can be quite large

    data.payload

FROM snowplow_failed_events.event_forwarding_failures

LIMIT 10
```

**4. Example queries**

Summarize the most common types of errors:

```sql
SELECT

    data.failure.errorType,

    data.failure.errorCode,

    -- time range for each - is the issue still happening?

    MIN(data.failure.timestamp) AS minTstamp,

    MAX(data.failure.timestamp) AS maxTstamp,

    -- How many errors overall,

    COUNT(*) AS errorCount,

    -- There might just be lots of different messages for the same error

    -- If this close to error count, the messages for a single error might just have high cardinality - worth checking the messages themselves

    -- If it's a low number, we might have more than one issue

    -- If it's 1, we have only one issue and the below message is shared by all

    COUNT(DISTINCT data.failure.errorMessage) AS distinctErrorMessages,

    -- a sample of error message. You may need to look at them individually to get the full picture

    MIN(data.failure.errorMessage) AS sampleErrorMessage

FROM snowplow_failed_events.event_forwarding_failures

GROUP BY 1, 2

ORDER BY errorCount DESC -- Most errors first

LIMIT 10
```

View transformation errors:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState,

    data.payload

FROM snowplow_failed_events.event_forwarding_failures

WHERE data.failure.errorType = 'transformation'

LIMIT 50
```

View API errors:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState

FROM snowplow_failed_events.event_forwarding_failures

WHERE data.failure.errorType = 'api'

LIMIT 50
```

Filter based on a date and hour:

```sql
-- Note that the times in the file paths are for the creation of the file, not the failure time



SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState, -- these are last because they can be quite large

    data.payload

FROM snowplow_failed_events.event_forwarding_failures

WHERE _FILE_NAME LIKE '%2025/07/29/16%' -- File paths are timestamped like this, so we can limit our queries this way

LIMIT 50
```

Filter for a range of timestamps:

```sql
SELECT

    data.failure.timestamp,

    data.failure.errorType,

    data.failure.errorCode,

    data.failure.errorMessage,

    data.processor.artifact,

    data.processor.version,

    data.failure.latestState, -- these are last because they can be quite large

    data.payload

FROM snowplow_failed_events.event_forwarding_failures

-- Here we need the full path prefix

WHERE _FILE_NAME > 'gs://{BUCKET}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/jsonschema-1/2025/07/29/16'

AND _FILE_NAME < 'gs://{BUCKET}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/jsonschema-1/2025/07/29/20'
```

***

---

# Configure Amplitude Tag for GTM Server Side
> Configure event mapping, user properties, entity rules, and session tracking for the Amplitude Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/amplitude-tag-for-gtm-ss/amplitude-tag-configuration/

> **Tip:** The [Session ID in Amplitude](https://help.amplitude.com/hc/en-us/articles/115002323627-Track-sessions-in-Amplitude) is the session's start time in milliseconds since epoch, so it cannot be derived directly from the `session_id` of your forwarded Snowplow events, which is a UUID. Therefore, in order to populate the Session ID so that your events are stitched into sessions correctly in Amplitude, your Snowplow events need to have the [`client_session` context entity](/docs/sources/web-trackers/tracking-events/session/) attached. Then the Amplitude Tag will automatically populate the Amplitude Session ID based on the `firstEventTimestamp` property of the session the event belongs to.

## Amplitude API Key (Required)

Set this to the API of your Amplitude HTTP API Data Source.

![Amplitude API Key section with an empty text input field and a Use Amplitude\&#39;s EU servers checkbox (unchecked).](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAATIAAAB5CAYAAAC6Lq3eAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABfTSURBVHic7d15XNTV/sfx1zDAsLth7kuIBgmaC6K4sIiSYtY1vZp6KyNxKZeyq2n200zJtXBL+91S07BLF70pIWi4XHfDrVQyQ36iqAgqm8Cwzfz+mOvoCCqb4lc+z8djHsJ3Oed8vzLvOed8vzOj0uv1eoQQQsHMqrsBQghRWRJkQgjFkyATQiieBJkQQvEkyIQQiidBJoRQPAkyIYTiSZAJIRRPgkwIoXgSZEIIxXvig+xmjo4mH1yr8nKD12ew4VBelZcrhHj8qiTI/rLyJjM2Z1dFUQ/18oqbxF0oBCAxrZjvDj/+MNr/ZwHNp14jNUtnsnzfnwU0+eCa8dFudhrvhWeRrTW8nXXtgVxe/yaj1DJfWXGTTce1xt93nc3HfVYa59OKHt2BCPGUqHSQJacXczaliMhftRQVV0WTHmzOy/a4NTEHIDGtiLBqCLJNx7XYalT8eEJbYl3LemqSFjYgaWEDtr9Xl+vZOj7ZWr6QP3u1iAkbs1g1shat6ptXVbOFeGpVOsg2H9cS6G6FU301O8/mm6xznZnKgphbdPo0DfdZafx4Qsv8bbdo81Eq3UKus//PAgDOphTRZe51xm7IxGPudQKX3iQxrfRUfC88i5MXC1l/MI8J32cSf6UQz3nXAVi9J5fJ/8wybjsnMps5kYYQySvU805YJi4zUxmyOp1rd/WminXwydZsXGem0mP+dXb+bnocd9MW6ok5pWXWS/ZEHCs9RM3NDI9GtdS84WXNiYuFZTiTBqnZOl7/JoPp/e3o0drSuPzExUL6fH4D15mpvBuWSW6BnuNJhbjOTDW+gOw9V0DXkOtlrkuIp0Wlg2zTMS2B7awY0M6KTaU8sfPy9ez+wJGPAu2YsDETjYWKX2fX560eNnz6052eSmp2MW/1sCFupiN9ntfwfnjmA+t93cua5a/V4vnGFhz5yPGh7VyxK4eUTB37pjnyYT87ElLvBOXq/+TwW3IRRz6qz9LXajHp+ywy83SllrP9TD6tnjFncGdrrmToOJty/6Hf9Vs6fjiax/ONy9ar0hbqGbUmg8D2GkZ2tTYuv5Wv5801GbzXx44Ts+pTqDMcT4fmFthoVBw8b3hB2PV7PgFtNWWqS4inSaWC7NdLhaRmF9OztSX922mIjS8gK8/0481GdrPGwVrFS+2t0OnhrR7WWFuo6NtWQ9KNO2FS28aMLs9aADDWx4ZjSYXGuaWqEHM6n3f8bKhvb0anFhZ0d7Ywrtt0TMu7frY4WKvo1MIC96bmHE4svRdlCG4N5mbQt62GTcdMh5cXbhQb58i8F97ATKVi9kD7MrVxYfQtzqYUYW6mMln+nz/yaVZXTX93DVYWKkb3tCE2Ph+VCl50s2LHGUMPcudZCTJRM1VqAmbTMS1922owV0OT2mpcG5mz9VetSW/CWJH6v//+90lqpjIM6UpjZaHCzkrF9Vs6almrSt+onK5n62haR13ququZxYzdkMHt/CjUwYD2JRt345aOvefymfsXQzANaKfh7xFZTO9vZ9y3ZT01B6Y/vIdYmucamhMyyJ6Xlt3Eu42lcWh5NVPH6cuGYSSAHnCwMrwG9XfX8N4/sxjdy4abOXq6OFner3ghnloVDrIiHWw5qeX6LR0RR+/0SizNKTXIyiNbqydbq8fRzozC4rL3yszVoNOVvn19ezNu5pSenA0c1Cz5qwOdWliUuv62LSe1FBZDt3vmoQ4mFJjMZ1XU0C7WOD9jztxBDkz8PpPYKfWoa2tGAwczvJwt2Ti6Tol9ujpZkleoZ/nOHPxdLTF/4m+oEaLqVfjPfu8f+ZipIGlhAy4vNjyOfuzI8aRCLt4s/+XLmzk6tp/Jp7BYz+c7btG5hQX2Vg/ujdlYGnptt4ezzzqqOZpUyPVbOn6/WkTUb3cm7V9007Bmfy7FOriSUcyxpDtDx5faa1gUc4v0XB3puYarjFczSx7DpmNa5v7F3ni8lxc3YGRXayKOlbx6WRmvdrSiu7Ml74cbLlz0bG3JqeQiYk7nU6SDPX8UsGpPDgBqM+jzvIZ/xuXRV4aVooaqcJBtOq5lqIe1SQ+gUS01vV01bK7AE9tOY0bUr1razU7jcGIhS4Y6PHSfTi0taOBghsfcNAB8XTR0bG6B12fXmfR9pvE2DYB3/GyxUKvwmJvG2A2ZNK97Z5g5wc+WNg3M8V10g57zb6A2U9HQwXQYmphmuM3k1Y6mvc3XvWzYdkpLXmHVfvVByCAH/kgpYt2BXGrbmLF2VG1CY28ZrgRH36Jn6zuh1d9dg4Vahc9zEmSiZlI9CV8+cjaliGFfpXNyVv3qbooiHTpfwKo9uawPql3dTRGiWsiMisKl5+pYtSeXVzpYVXdThKg2EmQK5zH3OjaWKl5+QYJM1FxPxNBSCCEqQ3pkQgjFkyATQiieBJkQQvEq9Ralk5eKCP35FrHx+chEmxDVb8+4qr05u6rY2lhTr25tNJpH8xa6CgfZyUtFDFh6Az2G903KJQMhql/rVi2quwmlysjM5mLyVZo3bfRIwqzCQRb68y30wFAPK+a/6oCledW8uVsIUX6P4uPgq1LtWoYPWrhxM4PGjZ6p8vIrPEcWG294r6WEmBCiLGrXsicn99F8onOFe2R6DMNJCTEhRHWTq5ZCCMWTIBNCKJ4EmRBC8STIhBCKJ0EmhFA8CTIhhOJJkAkhFO+xBpmTk9PjrE4IUUNIj0wIoXgSZEIIxavUx/gIIZQrIiKCwsI73+86ZMgQzM3N0Wq1/Pvf/zYu12g0DBo0CIC4uDgSEhKM6zw8PHB2dn58jb4PCTIhaqj8/HwKCgpKLNfr9Wi1pX+uWVFRkcm64uLyfxn3oyBBJkQNkpWVRW5uLmAIrLtdu3YNtVpNfn6+yXKdTkdKSgqAcd/bMjMzjeueeeYZzMyqZ7ZKgkyIGuTUqVOcO3eu1HU7duwodXlBQQHR0dGlrjtz5gxnzpwBYPjw4Wg01fNt9zLZL4Qolb29PebmyujrSJAJIUrl7e2No6NjdTejTCTIhBCKp4x+oxDisVCr1SV+v71Mp9OVuEDwpJAgE0IYDRw40Dhhr9Fo8PHxMd5i8csvv5CYmFidzbsvCTIhhNHdN8IOGDCAo0ePGm+veJLJHJkQQvEkyIQQiidDSyFEqaKjo9HpdNXdjDKRIBOiBnnuuedo3LjxIynbwsLikZRbFo81yJ7UKx5C1BSOjo6Kucm1PGSOTAiheBJkQgjFkyATQiieBJkQQvEkyIQQiidBJoRQvAoHmQowU0FB0ZP5bnghRM1R4SDzf16DTg8fbsqSMBNCPFRGZja2NtaPpGyVvoIfMHTyUhEDlt5Aj6FnppMsE6LaXV7coLqbUKqMzGzSrt+kedNGaDSWVV5+hYMMDGEW+vMtYuPzkRwTovrtGVf617hVN1sba+rVrf1IQgwqGWRCCPEkkKuWQgjFkyATQiieBJkQQvEkyIQQiidBJoRQPAkyIYTiSZAJIRRPgkwIoXgSZEIIxZMgE0IongSZEELxJMiEEIonQSaEUDwJMiGE4kmQCSEUT4JMCKF4EmRCCMUzr4pCTv52mqiYnykqLq5YI9RqAl/swwvt3KqiOUKIGqZKemSVCTGAouJiomJ+roqmCCFqoCrpkd0OsVnTP6jQ/p98trhSQSiEqNlkjkwIoXgSZEIIxVNskOXm5uLk5MThw4dNlp8/fx4nJyfOnz//SOotLi7Gw8ODuXPnPpLy77Z27VoCAgIAw3F5eXlx7dq1Kiu/Z8+ehIeHP3S7BQsWsHLlynKVPXPmTJycnEo8Jk6cCMDAgQNZunRpif26dOlCWFhYuep6mmRlZRF39DixO/ew/+BhLl+5Wt1NUoQqmSOrSQ4dOkR6ejo//fQTM2bMwMzs8bwWNGzYkKCgIOrWrQtAZGQks2fP5tixY4+8br1ej5WVVbn369y5M5999pnJMjs7u6pq1lMlKzub78MjOHwkDrVajYODPbm5eWi1Wp5t2YK/jRiG07Mtq7uZTywJsnKKjIxk8ODBREdHc+TIEbp16/ZY6rW1tSUoKOix1HUvvV5foQCysbGhVatWj6BFFafT6arsxaeqykpNS2PBolDUajXjx75Nh/btMDc3R6/X83//d4HNWyIJWbCEccFv0aljhypo+dNHsUPLsoqPj2fIkCG0bdsWX19fIiIiTNavWbMGLy8vOnbsyOTJk8nIyLhvWYWFhWzfvp2BAwfSu3dvIiMjTdYPHDiQuXPnMmzYMFxcXAgICOD06dOsXbsWDw8POnTowMyZM9HpdACEhITw5ptv8u677/LCCy/g5eXF5s2b73scTk5OpKenM2bMGCZNmkR6ejpOTk7s2rWLHTt20L59e5N9hgwZYjJ8+/bbb/Hw8KBLly4sWLCA4nuuFO/bt49+/frh5ubG0KFDOXfuHADe3t506GB4Av3000/4+/vTtm1bBg0axMmTJx90+qvUg+rOy8tjxowZtG/fnu7du7N06VLj8YWEhDB06FDeeustXF1dCQ8Pp0uXLsb/B4Dx48fz0Ucflbus/Px80tPTmThxIu3bt8fT05N58+aVOLf3U1RUROjyVTg42DPr4w9p6+rKP775lnET3mP2p4be7JTJE/Dx7slXX68l+fKVKjmXT5unPshGjx5Nu3btiImJYfz48cyYMYNff/0VgA0bNrBu3Tq++OILwsPDSU1NZdasWfcta8+ePZiZmeHp6cmLL75ITEwMRUVFJttERUUxbtw4IiIiqFOnDiNGjODQoUOsW7eOkJAQIiIiiI6ONm7/yy+/MHjwYA4ePMiYMWOYOnUqSUlJDzymZcuWsWjRIurUqcPp06fx9vZ+6HmIi4vj008/JTg4mPXr16PT6UhJSTGu//333xk3bhxjxowhKioKNzc3goKCKCwsxMvLizZt2pCYmMj777/PhAkTiI6Oxt3dnaCgIPLz8x9af2U9rO4PPviAy5cv869//YslS5YQHh5uMv937NgxfHx82LJlC/379ycrK4sTJ04Ahheoffv20b9//3KXZWlpyZw5c0hLS2Pz5s0sXbqUrVu38s0335TpuHb/Zx83btxk4jtjsbWx4buN/+Tcnwn89dW/UKuWA18s/xKtNp8Rw4bQrGlT/hXx76o6pU+VpzrIcnNzSUlJoVevXjRr1owhQ4awcOFCrK2tAfjHP/7B3//+dzw9PWndujXTp09n27ZtJcLptsjISPr06YNarcbb25uCggL27dtnss2IESPw9vbGzc2N0aNHk52dzbx582jbti39+vXDw8OD06dPG7fv0qULPj4+2NjY8MYbb+Dm5samTZseeFwajQZLS0vAMHxTq9UPPRdhYWH4+voyevRoXFxcmD59Og0aNDCuX7NmDa+88gqvvPIKLVq0YMaMGWRmZnL06FHjNklJSahUKvz8/GjevDnTpk1j0qRJaLXaUuvcv38/Li4uJo+CgoKHtrU0D6o7OTmZmJgYFi1aRJs2bejatSujRo1iy5Ytxv09PT15/fXXcXFxwd7enp49e/Lzz4absA8ePIilpSVdu3Ytd1kqlYrExEQ6dOhAq1at6Nq1K6Ghobi4uJTpuA4ficOrmyd16tQG4PKVq7w2dDC+Pr0IDhpFdvYtLiUno1KpCOwXwKkz8eTk5FToHD7Nnuo5MhsbG4KDgxk3bhw+Pj74+fnRv39/bGxsyMrKIjk5mWnTpjF9+nTAMBdUXFzM1atXadasmUlZubm57Ny5kxUrVgCGMPH19WXr1q34+voatzM3v3NKbW1tAYwT9LfblJube982u7q6cvHixcof/D0uXLiAv7+/ybK72xofH09CQgI//vijcVleXh6XL182/u7l5UWnTp3w9fUlICAAf39/Ro4ced95oo4dO5aY7LewsKhQ+x9U95EjR9Dr9fj5+Rm3LyoqwtHR8b71BgYGsmzZMj788EN27txJ3759UavVxMfHl7us8ePHM2XKFI4fP46fnx8DBw6kYcOGZTqu5MtX6O3nY/z9k/+ZYfx53/6D2NvZ0bRJEwCcnZ3Q6XRcTbmGcyunMpVfUyg2yCwsLDAzM6OwsNBk+e1XfI1GA8C0adMYNmwYsbGxhIWFsXjxYn744Qdq1za8An7++ee4urqalNGoUaMS9cXGxpKXl8fYsWNRqVSAYbLX0tISrVZboat6pbn3eKqKSqUy9uLuZ9SoUbz22msmy+rVq2f8WaPRsHHjRo4fP87u3bv5+OOPadKkCWFhYSaheNuDJvs1Gk2pvbOCggLj/92929+vbgBLS0uioqJM9nlQT7VPnz5Mnz6dP//8k9jYWObPn29cV96yAgIC8PT0JDY2ltjYWEJDQwkNDaVv37733ee2oqJCLEsJ95+2xbAlchtTp0zCxsba2C6gwr3ap5lih5YWFhY0a9bMZJgGcPr0aaytrWncuDEJCQksWbKE5s2bExQUxObNm6lVqxZbt27FwcGB+vXrk5ycTIsWLYwPR0fHUp+UkZGRDBgwgG3bthEVFWV8qFQqdu3aVeHjuHvCGeDUqVO0aNHiofvdDtPb7OzsyMvLMynv7lB89tln+eOPP0z20ev1xp+dnZ1JSEgocS7uvlq5e/dufvjhBzp27MiUKVPYsmULcXFxFZrwd3Z2LvF/d/HiRbKyskoNvwfV7ezsTEFBAdnZ2ca2N23a1KQXdS9bW1t69erFF198gVarxcvLy9iu8pRVUFDAwoULycnJYfDgwaxevZrBgwezfv36Mp2H2rVrk1LKvYFR0Tt4+aX+tHa+cy6upRi2u7uHLwwUG2QAb7/9NqtWrSI8PJyzZ88SGRnJwoULCQoKwszMDDs7O77++mtCQ0O5dOkSBw4cICUlBScnQ7c8ODiY5cuXExkZyaVLl1i5ciWDBw82eYIDZGZmsnfvXgYNGkSrVq2Mj9atW5d69bI8Dh8+TEREBBcvXmTJkiUkJiby6quvPnS/OnXqkJGRwYEDB8jIyKB169aoVCq+/PJLzp07x+rVq/ntt9+M2w8fPpxt27axc+dO8vLyCAsL48qVO1fA3n77bfbs2cPy5ctJSkpi+/bt9OrVy+SCQEFBAbNmzSIqKorLly8TFRWFubl5iWH4bXl5eSQlJZk8bt/Q+8Ybb3Do0CEWLVrEmTNnOHDgABMmTKBz587GK6R3e1DdTk5O9O7dmylTphAXF8f58+eZOHFiiWHtvQIDA4mJiaFPnz7GF6/ylmVpacnu3buZPXs2CQkJxMfHc/LkSePf2MO0c2vLocO/mPzN6fV63vzbcHp4md7ac+DwERwd69GwwTNlKrsmUezQEgxPTpVKxZo1a7hw4QKNGzcmKCiI4OBgwHAT6ddff838+fP56quvqFevHsHBwcarU6NGjSInJ4d58+aRkZGBu7s7S5YsKdHbiYmJwcrKiu7du5doQ2BgIBMmTCA7O7tCx9CuXTsOHTrEvHnzsLa2NvYgH8bT05Nu3boRHBzMsmXL6N27N3PmzCE0NJQ1a9bg5+dH586djdt37tyZadOmMW3aNPR6PX379sXZ2dm43t3dnRUrVvD555+zfPlymjdvzuzZs03megICApg6dSoLFiwgNTWVli1bsnLlSpOLBneLi4szmT8Ew1zXd999h4uLCxs2bGDx4sWsX78eKysr/P39mTFjRqllPazuxYsX88knnxAUFGSc45o8efIDz6G/vz9WVlb069fPZHl5y/rqq6+YPXs2L7/8MhqNBj8/P6ZOnfrAum97sa8/e/cfJDIqhoEDDO3Q6fWs+t9vmDxxPC/Udgcg4Xwiu3bvZeTwoWUqt6ZR6e/tflTAvIVfVPrTK8zVaj6a+l5lm6IoISEhnDt3jnXr1lV3U0Q12rN3P99u2MiA/gG8/FJgiamNY8dP8PXa9bR93pV3xo4u8UIrqqhHFvhinyr5YEUhaiKfXj0wV6vZsDGcfQcO0aF9O+rWrUtuXi7x8We5eCkZn149GDl8qITYfVRJkL3Qzk0+3VWISujRvRvubm3Zs3cfv589xx/n/sTK2hpnZyfefH0Ez7Z8+AWgmqxKhpZCCFGdFH3VUgghQIJMCPEUkCATQiieBJkQQvEkyIQQiidBJoRQPAkyIYTiSZAJIRRPgkwIoXgSZEIIxZMgE0IongSZEELxJMiEEIonQSaEUDwJMiGE4v0/o6b4azBd12AAAAAASUVORK5CYII=)

### Use Amplitude's EU servers

Enable this option to send the data to Amplitude's EU Residency Server [endpoint](https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/#endpoints), instead of the default standard server endpoint.

## Snowplow Event Mapping Options

### Include Self Describing event

Indicates if a Snowplow Self Describing event should be in the `event_properties` object of the Amplitude event.

### Snowplow Event Context Rules

This section describes how the Amplitude tag will use the context Entities attached to a Snowplow Event.

![Snowplow Event Context Rules section with Extract entity from Array if single element checked, Include Snowplow Entities in event\_properties set to All, an empty Snowplow Entities to Add/Edit mapping table with Entity Name, Amplitude Mapped Name, Include in event\_properties or user\_properties, and Apply to all versions columns, and an empty Snowplow Entities to Exclude table.](/assets/images/02-gtm-ss-amplitude-6f8f90af92e7fefe68e952297bed66cd.png)

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in event\_properties

Using this drop-down menu you can specify whether you want to Include `All` or `None` of the Snowplow context entities in Amplitude's `event_properties`.

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- The Entity name to add/edit-mapping (required).¹
- The key you could like to map it to (optional: leaving the mapped key blank keeps the same name).
- Whether to add in `event_properties` or `user_properties` of the Amplitude event (default value is `event_properties`).
- Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if `Include Snowplow Entities in event_properties` is set to `All`), you can specify the context entities you want to exclude from the Amplitude event. In its columns you can specify:

- The Entity name (required).¹
- Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

**pre-v0.2.0**

#### Snowplow Event Context Rules

##### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

##### Include all Entities in event\_properties

Leaving this option enabled ensures that all Entities on an event will be included within the Event Properties of the Amplitude event.

If disabling this, individual entities can be selected for inclusion. These entities can also be remapped to have different names in the Amplitude event, and can be included in either event properties or user properties. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1` where `com_snowplowanalytics_snowplow` is the event vendor, `web_page` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0`

##### Include unmapped entities in event\_properties

If remapping or moving some entities to User Properties with the above customization, you may wish to ensure all unmapped entities are still included in the event. Enabling this option will ensure that all entities are mapped into the Amplitude event.

## Additional Event Mapping Options

If you wish to map other properties from a Client event into an Amplitude event they can be specified in this section.

![Additional Event Mapping Options section showing Event Property Rules (Include common event properties checked, empty Additional Event Property Mapping Rules table) and User Property Rules (Include common user properties checked, Map Snowplow mkt fields to user properties unchecked, empty Additional User Property Mapping Rules table).](/assets/images/03-gtm-ss-amplitude-124150b9c2edaf9f44f0ae0067a19c0d.png)

### Event Property Rules

#### Include common event properties

Enabling this ensures properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Amplitude Event Properties.

#### Additional Event Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client. These keys will populate the Amplitude `eventProperties` object.

### User Property Rules

#### Include common user properties

Enabling this ensures user\_data properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Amplitude Event Properties.

#### Map Snowplow mkt fields (standard UTM parameters) to user properties

Enabling this option automatically maps all the marketing (`mkt_` prefixed) fields of the Snowplow event to the standard UTM parameters in Amplitude's user properties.

#### Additional User Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client. These keys will populate the Amplitude `eventProperties` object.

### Groups Property Rules

> **Note:** This configuration option is relevant **only if** you have set up [account-level reporting in Amplitude](https://help.amplitude.com/hc/en-us/articles/115001765532).

#### Groups Property Mapping Rules

Specify the Property Key from the GTM Event, and the key you would like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0). These keys will populate the Amplitude `groups` object.

## Additional Properties

In this section you can additionally set specified event or user properties to custom values (e.g. through a GTM Server-side variable).

![Additional Properties section with three empty tables for setting custom values: Event Properties (Amplitude Property Name and Value columns), User Properties, and Groups Properties, each with an Add Row button.](/assets/images/05-gtm-ss-amplitude-e77fabbf0070687d579898d733c60e33.png)

### Event Properties

Using the **Additional Event Properties** table allows you to set additional **event** properties in Amplitude payload. To do so add a row and simply specify the property name for Amplitude `event_properties` and then the value you would like to set it to.

### User Properties

Using the **Additional User Properties** table allows you to set additional **user** properties in Amplitude payload. Similarly to the previous table in the section, add a row and specify the property name for Amplitude `user_properties` and then the value you would like to set it to.

### Groups Properties

> **Note:** This configuration option is relevant **only if** you have set up [account-level reporting in Amplitude](https://help.amplitude.com/hc/en-us/articles/115001765532).

Using the **Additional Groups Properties** table allows you to set additional **groups** properties in Amplitude payload. Similarly to the previous tables in the section, add a row and specify the property name for Amplitude `groups` object and then the value you would like to set it to.

## Advanced Event Settings

In this section you can find advanced configuration parameters.

![Advanced Event Settings section showing Forward Users IP address checked, Fallback platform identifier set to Web, Amplitude event time set to Do not set, Inherit Amplitude device\_id from common event client\_id checked, and Inherit Amplitude user\_id from common event user\_id checked.](/assets/images/04-gtm-ss-amplitude-5cf1cc255705dea76631842bde590aa6.png)

### Forward User IP address

Enabling this will forward the IP Address to Amplitude, otherwise Amplitude will not receive the users IP Address (default: `True`).

### Fallback platform identifier

If there is no Platform property on the Client event, this is the value which the Tag will forward to Amplitude (default: `Web`).

### Amplitude time setting

This option allows you to decide whether the event time of the Amplitude event will be set. The available options are:

- `Do not set` (default): this means the event time will be set automatically by Amplitude.
- `Set to current timestamp`: sets the Amplitude event time to the current timestamp.
- `Set from event property`: sets the Amplitude event time from the client event property. For example, in the image below the Amplitude's event time will be set from the device created timestamp (`dvce_created_tstamp`) of the Snowplow event (`x-sp-` prefix in the client event):

![Amplitude time setting section with Amplitude event time set to Set from event property and Event property name field containing x-sp-dvce\_created\_tstamp.](/assets/images/07-gtm-ss-amplitude-dda2f8b6d3e860dc2e949e845b21440b.png)

### Device Identifier

#### Inherit Amplitude `device_id` from common event `client_id`

By default the Amplitude tag sets the `device_id` property of the Amplitude event from the `client_id` property of the common event. Unchecking this tick box allows you to override the value for `device_id` in Amplitude event payload.

### User Identifier

#### Inherit Amplitude `user_id` from common event `user_id`

By default the Amplitude tag sets the `user_id` property of the Amplitude event from the `user_id` property of the common event. Unchecking this tick box allows you to override the value for `user_id` in Amplitude event payload.

## Logs Settings

Through the Logs Settings you can control the logging behavior of the Amplitude Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Amplitude GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{

    "Name": "Amplitude HTTP API V2", // the name of the tag

    "Type": "Message",   // the type of log (one of "Message", "Request", "Response")

    "TraceId": "xxx",    // the "trace-id" header if exists

    "EventName": "xxx"   // the name of the event the tag fired at

}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# Amplitude Tag for GTM Server Side
> Forward Snowplow events to Amplitude from GTM Server Side using the Amplitude Tag with HTTP API v2 for product analytics and user tracking.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/amplitude-tag-for-gtm-ss/

The [Amplitude Tag for GTM SS](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-server-side-amplitude-tag) allows events to be forwarded to Amplitude. This Tag works best with events from the Snowplow Client, but can also construct Amplitude events from other GTM SS Clients such as GAv4.

The tag is designed to work best with Self Describing Events, and atomic events, from a Snowplow Tracker, allowing for events to automatically merged into an Amplitude events properties. Additionally, any other client event properties can be included within the event properties or user properties of the Amplitude event.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click “Search Gallery” in the Tag Templates section
2. Search for “Amplitude HTTP API V2” and select the official “By Snowplow” tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking “Add”

## Amplitude Tag Setup

With the template installed, you can now add the Amplitude Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the Amplitude Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to Amplitude
3. Enter your Amplitude API Key for a HTTP API integration. This can be retrieved from Amplitude Data Sources within your Amplitude project.
4. Click Save

---

# Configure Braze Tag for GTM Server Side
> Configure authentication, user identifiers, event mapping, and entity rules for the Braze Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/braze-tag-configuration/

## Configuration options

### Braze REST API Endpoint (required)

Set this to the URL of your Braze REST [endpoint](https://www.braze.com/docs/api/basics/#endpoints).

### Braze API Key (required)

Set this to your Braze [API Key](https://www.braze.com/docs/api/basics/#app-group-rest-api-keys) that will be included in each request. The minimum permission that you need to assign for this API Key is to access the `/users/track` endpoint.

![The Braze API key permissions list showing \&quot;users.track\&quot; checked as the required permission for recording user attributes, custom events, and purchases](/assets/images/key_permission-0327a99d1c82d569bcce97d5ae5508e4.png)

### Identity settings

#### Braze User Identifier

This section allows you to select which Braze user identifier (external user ID (`external_id`) or [User Alias](https://www.braze.com/docs/api/objects_filters/user_alias_object#user-alias-object-specification) `user_alias`) will be used by the tag. The default value is `external_id`.

![The Identity settings section showing the Braze User Identifier dropdown open with \&quot;external\_id\&quot; selected and \&quot;user alias object\&quot; as the alternative option](/assets/images/identity_settings-bf117a2d0fb59b9460f358912d794ac5.png)

##### Braze external\_id

This configuration section is enabled if you have selected the `external_id` as the **Braze User Identifier**.

![The Braze External Id section with \&quot;Set external\_id from:\&quot; set to \&quot;Event Property\&quot; and an empty field for the external\_id value](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAe8AAADHCAYAAAAwAdsiAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d15eFTl3f/xdxaykpiFEJIgoSGBBBIi+74GAgSw0UcNKikqaLW1rVWqQFv118dKXWsrVikqKvWxigsKsq+iyBZBgQABAiSENStZJzM58/uDZiRNgCQGJ0c/r+uai5lzzn2f70mu8Jn7PmfOuJSUltn5gbHb7fWeG4bh+Ndut2MYBjU2GzWGgc1mczwsFgtWq5V+fXo7pXYREZErcXV2ASIiItI0Cm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTcXd2ASIAx44d4x//+IfjtYeHB+3bt2f06NHEx8c7sbJvzZo1y3Gb3YtNmDCBUaNGOaGiptm/fz8LFy7klltuoW/fvvXWv/POO+zatYs//vGP+Pn51Vu/fft23n//fe6++25iYmK+j5JF5BIU3tKqxMTEEBUVRXV1Nbt372bRokX88pe/pFOnTs4uDYDg4OB6wRcVFXXFduvWrWPVqlU88MADhIeHX63yRORHQuEt9Rw7doxDhw4xduzYBtevWbOGmJgYOnfu3OL7joqKIikpCYDu3bvz0ksvkZmZ2WB4G4aBq+v3e+YnMDDQUd/3yRnHKiKtl8Jb6lm3bh0HDx6kpqaG8ePH11m3YsUKNmzYQG5uLnfddddVrcPHxwcAi8UCwNatW/nwww8ZOnQou3fvZujQoYwePZpdu3axfv16CgoKCAwMJDk5mcTERADmzp1LUVFRvX4ff/xxADIzM1m5ciWFhYWEhoYyceLERo2k/9uWLVtYsmQJN910E/3792fXrl288847TJgwgZycHPbt2wfACy+8wLhx40hKSsJut7Nx40a2bNlCdXU1nTt35oYbbiAgIKDBY+3evTvPP/88I0aMIDc3l5ycHDp06EBaWhqhoaEAnDhxgmXLlpGTk4OPjw/9+vUjOTkZFxeXJh9TZWUlixcv5uDBgwQHB3Pttdc2uQ8RuTr0Vl7qSU9Pp3Pnzqxfv56VK1c6ltcGd+fOnZk6depV2bfFYqG0tJT8/HxWrVqFi4uLI4hr7dixg5iYGCIiIsjOzuadd94hICCAG2+8EV9fX9555x3Onj0LQHJyMqmpqaSmptKrVy8AunbtCkB2djZvvvkmfn5+JCcnYxgGr7/+OufPn79kfTabjaKiIsejdtuBAwcSGhrK2rVrqaqqYuXKlQQFBTFs2DBGjRrl2PfkyZPp2bMncOFN0ooVK4iLi2PkyJHk5OTw1ltv1flK24uPtdbmzZvp0KEDAwcOJC8vj08//RSAqqoqFixYwPnz50lNTaVbt26sW7eOHTt2NOt38emnn7J371569OhBbGws33zzTbP6EZGWp5G31OPh4cGMGTN49dVXWb9+PXBh2nbjxo107tyZGTNm4OHhcVX2vXHjRjZu3Oh4PWDAgHrT86mpqfTp0weAgoICpk2bRlRUFN7e3gQFBfHKK69w7Ngx2rdv79iuqqqKTZs24e/vT2pqKgCbNm3C19eX22+/HXd3d6Kiovj73//Onj17GDJkSIP1HTt2jLlz5zpeh4aG8tBDD+Hq6srkyZN59dVXeeWVVygqKiI9PR13d3c6derEoUOHAOjSpQshISEYhsHmzZvp3r07kyZNAsDd3Z2lS5dy5syZBo/19OnTAAwZMoTJkycDkJOTw4kTJ4AL311/yy23EBoaSrt27ejbty8ZGRkcOXKE/v37N+n3YLPZyMjIICoqittuuw2Atm3bsmzZsib1IyJXh8JbGtRQgF/t4Abo06cP1113HYZhcODAAb788kvCwsIYPHiwYxt/f3/H8+DgYLKzs3n11VcpLCykuroa+HaqvdbHH39McXEx06dPd0zHnzx5krKyMscUeq3/nma/WFhYGMnJyY7Xnp6ejuddu3YlNjaWAwcOEBUVRUJCwiX7KSoqorKykszMTP7whz/UWVdYWNjgsdaqrR/Ay8vLccze3t74+PiwZMkSzpw5Q2VlJYZhONY3RVFRETU1NfzkJz+psy8RaR0U3nJJFwc4cNWDG6Bdu3Z069YNgNjYWL7++mt27txZJ7wvtmfPHhYvXszgwYO56aabyM/PZ9GiRXW22bt3LxkZGQwePNgxZV4rODiYtLS0OssaCsxavr6+9OjR45LrKysrHf/a7fYrnmtOSEhg2LBhdZa1b9++WVPUxcXFvPbaa3Ts2JGpU6fStm1bnn322Sb3AzgujnNzc2tWexG5uhTeclm1AV77/PtktVqx2Wy0adPmktscPnwYgLFjx+Lr60tpaWmd9WVlZXzwwQeEhIQwceLEOutCQ0M5cuQIAQEBBAQEAJCbm0tQUFCz6t29ezfHjx93jL63b9/OgAED6mxTez47MDAQDw8Pzp49S2RkJC4uLtjtdvLy8uqMrJvi+PHjVFdXM2TIECIjI5s14q4VEBCAu7s7p06dciyz2WzN7k9EWpbCW67o+wzto0ePsmHDBmpqasjMzKS6uvqyN2kJCQkB4MMPP6Rjx458/vnnAI6bqbz//vuUl5cTHx9PRkaGo12/fv0YPXo0WVlZ/POf/6R///4UFBSwfft27r333jrTxRcrLi5mw4YNdZZ17tyZjh07snz5ckJCQpg2bRrz5s1j5cqVJCYm4uXl5RjNb9iwgYEDBxIdHc3IkSNZvXo1CxYsIDY2loMHD5Kbm1tvGr2xQkJCcHFxYf369RQXF7Nr1y5qamoavLHMlbi5udG3b1+2bdvGmjVr8PPzY+3atc2qS0RansJbWpWsrCyysrJwc3MjKCiIlJQUhg4desntBw0axNmzZ/n66685ceIEw4YNY/ny5Y7zxtnZ2QBs27atTrvExESioqJIT09n9erVrFq1Cn9/fyZOnHjJ4AbIz89nxYoVdZYlJSVx+PBhiouLSU9Px83NjZSUFBYsWMCaNWscV5hnZGRw4MABOnbsSHR0NKNHj8Zut7N9+3aOHj1Khw4dmDp1arPfLIWHh3PDDTewbt061q1bR79+/aioqKhzDr0pUlJSKCsrY8OGDQQGBtKzZ0+2bNnSrL5EpGW5lJSW2a+8mblc/FGb2ue1ow/DMLDb7RiGQY3NRo1hYLPZHA+LxYLVaqVfn95OqV1ERORK9DlvERERk1F4i4iImIzCW0RExGQU3iIiIiaj8BYRETEZhbeIiIjJKLxFRERMRuEtIiJiMgpvERERk1F4i4iImIzCW0RExGQU3iIiIiaj8BYRETEZhbeIiIjJKLxFRERMRuEtIiJiMgpvERERk1F4i4iImIzCW0RExGQU3iIiIibj7uwCROTHx2KxcDDrEPn5Bbi5uREW1oHoLlG4umo8IdIYCm8R+d7UGAZLly1nxaq1VFdXc801/tTYaigtKyMwIICbb7qBwQP7O7tMkVZP4S3iBEuWLMHX15exY8c2q/2iRYuIiYlh4MCB9daVlZWxaNEi8vLy+O1vf0twcPB3LbdFWK1W/vr3f5B99Bg3/HQSw4YMwtfXF4D8/AJWrl7LgtfeIDf3BGk33+jkakVaN4W3OM2RI0dYvnw5p0+fxs/Pj+HDhzN48OArttu2bRtRUVGEhIS0SB3nzp0jOzubAQMGtEh/LeWll15i4sSJdO7cuUnttmzZgo+PD4899hju7q3nT3zR2//meE4uf5g1k4iIcD76eBkbNn2GRxsPUn86iam3pdE1JpqX//kaYR1CGT5siLNLFmm1Ws9ftvyoVFRU8MYbb3DTTTfRvXt3zp49y5tvvomfnx8JCQmXbbtt2zb8/f1bLLzz8/PZtm1bk8LbbrcD4OLi0iI1NOSnP/0poaGhTW5XWFhIVFQUbdq0qbfObrdf1ZovJTf3BJu/+JJf/eLndOwYwWebv+DTFatIvX4SpWWlvP7GIsI6hNK/Xx+OHc9h8YdLGDigHx4eHt97rSJmoPAWpygoKMDV1ZXExEQAIiIiGD9+POXl5QAYhsGnn37Kjh07aNu2LZMnTyYuLo6//OUvFBcX8+9//5tx48bVG6lfqt3Zs2d58cUXeeihhwgICGDRokX4+/vTvn17Vq5cicViYe7cucyePRuAnTt3smrVKgzDYPDgwSQlJQHw+OOPExsby759+3jggQc4ePAgR44cwTAMDh8+TKdOnUhPT8fLywubzcZHH33E3r178fLyYuLEifTs2bPRP6N3332X1NRUunTpgtVqZfHixRw4cICIiAhsNluDbZYsWcI333zDvn37OHjwIDNmzGDRokXY7XaOHz/OuHHj6N+/P9u3b2ft2rVYrVYSEhKYPHkybdq0YfPmzRw6dIiamhqOHz9OZGQk48ePZ/HixRQWFtK9e3emTJmCq6srn3zyCT4+PowZM+aKx7J1+046dAild68Lv++jx3MYmzSKyRPHA5CZeYDM/QeJ7hJFyvhkVq5ey77MA/S6rvE/L5EfE13aKU4RFhZG27Ztef/99yksLASgd+/ejnO4n332GXl5ecyePZu0tDTeffddKisrmTVrFuHh4UyZMqXBKfZLtWvfvj0DBgxgxYoVHDt2jKNHj5KcnMygQYOYMmUK4eHhjuDOyclh+fLl3H333fzmN78hIyODw4cPO/bh7e3N7NmzCQoKAuDgwYOMGDGCOXPmUFVVxa5duwBYsWIFVquVRx99lJtuuol33333kqF7JRs2bOD8+fM8/PDDjB8/nnPnzjW4XWpqKgkJCSQnJzNjxgzH8rKyMn71q1/Rt29fsrOzWblyJXfeeSePPPIIxcXFrFq1yrFtbm4uKSkpzJo1i7KyMt5++22mTZvGww8/TG5uLgcOHACgf//+9OrVq1H1n8jLIya6i+P1tKm3MuWW/wEg69Bhzp47R2y3GADatvWlQ2h7TuSdbNoPSeRHROEtTuHu7s4vf/lLvLy8mDdvHi+99JIjFAAyMjIYNWoU3t7eREZGEhERQXZ29hX7vVy7sWPHkp2dzXvvvUdKSgre3t4N9vHVV1/Rt29f2rdvj7+/P3379mX//v2O9YMGDcLHx8cx/RwXF0fnzp3x9vYmKiqKgoICAIYMGUJqaipubm7ExMTg4uJCUVFRs35ee/fuZeTIkbRt25bIyEi6dOly5UYX6d27NwEBAbi6urJr1y769+9PWFgYXl5eTJgwgYyMDMe20dHRRERE0LZtW2JiYujevTvBwcH4+/sTGRnpOL4OHTo0+mI4q9WGRwPT+AcOZvH0c3/jxtTr6RoT7Vju4eGB1VrdpGMU+THRtLk4jY+PD5MmTSIlJYV9+/bx3nvvceONNxIfH8/58+f517/+5QhIwzAaNeV8uXaenp706dOHL7/8kt69e1+yj5KSErKysti6dStw4Txxjx49GnVMLi4u1NTUABfeoCxdupSTJy+MIK1Wq+NceVOVlZURGBjYrLb/rbS0lIiICMfrwMBAysvLsVqt9bb9789du7i4YBhGk/cZFBjA6TNn6i1fvWY93brGMD7526l3wzA4dy7fMbMhIvUpvMUpvvnmG4qKihgxYgSurq4kJCSQl5fHvn37iI+Px8/Pj5tvvpnIyMgm9Xu5dhUVFWzfvp3g4GC2bt16ySvb/f39GTNmDKNGjWrWsdX68MMP6dSpEzfffDOurq48+uijze7Lz8/PcT3AdxUQEEBJSYnjdVFREV5eXg1e4NZSEuJ7sOD1NyksLCIo6Ns3IWOSLswmXGz3N3uoqKykR/fYq1aPiNlp2lycIjAwkHXr1pGZmYnVaqWwsJCDBw8SFhYGQGJiIqtXr6aiooKKigqWLl3qCBwPDw8KCgocI9yLXa7d8uXLSUhIIC0tjdWrV1NWVubor6ysjMrKSgB69uzJli1bOHPmDFarlc8//5ysrKwmH2NZWRkuLi5YLBa2bduGxWJp9si7R48efPHFFxiGQXFxMTk5Oc3qBy5MoW/fvp0zZ85QXV3N6tWrG33u+mJnzpxxXK9wJX379iY4OIiFb/2rzsj9o4+XseXLbY7XpWVl/N+/FzOgX19C2rVrck0iPxYKb3GKa6+9lltvvZW1a9fy2GOP8fLLLxMTE8PQoUMBGDVqFKGhoTz77LM8/fTTuLq64u/vD1y4UGrVqlV8/vnn9fq9VLucnBz27NlDcnIyoaGh9OzZk2XLlgEQGRmJn58fTz75JABdunRhzJgxLFy4kD/96U8cOXKEjh07NvkYJ02axM6dO5k7dy4nT56kU6dOjjcMTTVq1Cjc3Nx48sknefvtt7/TFHqnTp1ISUnhjTfe4IknnnBcCd9UX375JV999VWjtnVzdeUXP59B1qEj/G3ey5wvLQXg97NmOi5cO3Eij7lPPYebqyvpt6c1uR6RHxOXktKy5g0FWrGLRze1z2vf7RuGgd1uxzAMamw2agwDm83meFgsFqxWK/36XPqcqEhLyc3NZf78+fWWP/jggz/Ic75Hjx1n3sv/pKysnOsSE4gID8Nmq+HosWPs3bef6C5R3H/fPVxzjb+zSxVp1RTeCm+R75XVauXzL77k6z17yS8oxM3VlbAOHejfrw+9ruvplJvIiJiNwlvhLSIiJqNz3iIiIiaj8BYRETEZhbeIiIjJKLxFRERMRndYE6exWKopKCymvKLS2aWISCvn6+NNcFAAnp76mlhQeIuTWCzV5Jw4RUi7IMLD2ju7HBFp5YpLSsk5cYpOHcMU4GjaXJykoLCYkHZBBFzj5+xSRMQEAq7xI6RdEAWFxc4upVVQeItTlFdUKrhFpEkCrvHTabb/UHiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZPTFJGJqY8aMITs7u97y//3f/+X22293QkVw7tw5BgwY4Hjt7u5OZGQk6enppKen4+Li4pS6ROSHQ+Etpjd9+nSmTJlSZ1lISMhV219tOK9cuZKuXbtecrsXXniB2NhYampq2L17N3/+858xDIM77rjjqtXWFI09DhFpfRTeYnrBwcF06dLF2WXUc+211zpCMS4ujoKCAl599dV64W0YBq6u398ZrJqaGtzc3L63/YlIy9M5b/nBGjlyJIsWLXK8zsrKokuXLpw+fRqAzZs3M2HCBOLj40lLSyMrKwuAiooKoqKiePfdd5kwYQI9evRg2rRpFBYWsnDhQseU+Pjx43nyyScbXU98fDwnT57EarXy5JNPkpaWxl133UVcXBwWi4WKigpmz57NddddR2JiIo888ggVFRV1apo3bx5JSUnExcUxffp0ioqKHP1XVlYyZ84cEhMTGTJkCH/729+oqakB4Prrr2fmzJlMmDCBtLS0Bo/jvvvu495773X0V1JSQkxMDBs3bmzGT19+zMrLy6murqaqqgqLxVLnUVVVRXV1NeXl5c4u09Q08pYfrJSUFFavXk16ejoAa9eupXfv3nTo0IH9+/dz33338cQTT9CrVy/eeustpk+fzvr16x3t58+fz5///Gfc3d158MEHmT9/Pr/73e9ISkpi5MiRLFmyhLi4uEbXc/r0aQICAmjTpg0AGRkZPProozz88MN4eHhwzz33cPr0ad58800Mw2DOnDn8/ve/569//aujj3379rF48WLOnz/PL37xC5544gmee+45AGbOnElZWRmLFy+msLCQBx98kJCQEG677TYAPvvsM+bOnUtMTAzh4eH1jmPNmjXMnDmTyspKvL29+eyzz/D19WXo0KHf+XchPy4ZGRmsWbMGV1dXDMOos6522dixYxk+fLiTKjQ/jbzF9J5//nliY2Mdj9rz3ykpKWzbto3S0lLgQninpKQA8Prrr5OamkpqaiqRkZHMmTOHkpISdu7c6ej3scceY9CgQfTr14/rr7+effv24e7ujre3NwBeXl6OIL4cu93OgQMHmD9/PhMmTHAsHzBgAD/72c+IjY3l2LFjrFu3jqeffprExER69erFM888wyeffEJeXp6jzT333ENQUBCdO3fm4YcfZunSpVgsFk6cOMHKlSt55pln6Nq1KwMHDuTOO+/k448/drSdOnUqSUlJdOrUqcHjGDVqFK6urmzevBmADRs2MGbMGNzd9R5fmmb48OEMHToUwzDqjbwNw2Do0KEK7u9If5VietOmTatzwZqXlxdwYZo6IiKCDRs2MGjQIPbs2cNLL70EQGZmJocPH2bJkiWOdpWVleTl5ZGYmAiAv7+/Y523t7djCruxpkyZgouLC4ZhYLfbmThxIrNmzXKsvzj49+/fj4+PT52RfHx8PF5eXhw6dIj+/fvX6z82NhabzcapU6fIysrCbrczevRox3qbzUa7du0crz08PC5br7e3N0lJSaxZs4YxY8awadMmnn766SYdswhceMM6btw4XF1d2bhxo+P0jZubG8OHD2fs2LHY7XZ98uI7UHiL6V3ugrWUlBTWrFlDRUUFPXv2JCwszLHuzjvv5NZbb63XV0t57rnniIuLw93dnfDw8MuO0j08POpdtGa32zEMg+rq6gbbWK1Wx3a1fXz66ad1tmnqhWkTJ05k9uzZfPXVV1RXVzNs2LAmtRcBcHFxwW63O0J606ZNAIwYMULB3UIU3vKDNnHiRNLS0jh//rxjyhwgOjqaw4cPExkZ6VhWXl6Or6/vFUfYjf1PJyIiotFXwcfExFBWVsbhw4eJjo4GYO/evVgsFmJjYx3b1QZ17fraNwZ2u53q6mpKS0uJj48HLlxVXhvwjT2OkSNHYrVaeeaZZ0hKSrriaF3kUmoDPDk52THyTk5OVnC3EJ3zFtMrKiri+PHjdR4lJSUAdO/enfbt27N582bGjx/vaDNjxgw2btzIiy++yPHjx1m1ahXDhw93XIl+Of7+/ri6urJ+/fpGbd8YkZGRjB8/npkzZ/LNN9/w9ddfM2vWLJKTk+nUqZNju+eee46srCx27tzJU089xQ033ICnpydRUVEkJSXx0EMPsWPHDo4cOcKvf/1r5s6d26Tj8PDwIDk5mR07dtR5syPSHLUBPmHCBCZMmKDgbkEKbzG9BQsWMGrUqDqPiz8iNnHiRHr27ElERIRjWUJCAvPmzWPp0qWMHTuWZ555hscff5wOHTpccX8eHh78/Oc/Z968ebz55pstdhxPPfUU0dHR3H777dxxxx0kJiby/PPP19mmd+/e3H///aSnpxMdHc2cOXMc65599ll69OjB9OnTSU1Nxd3dnQceeKDJxzFu3Dh8fHx0QZG0iNoAV3C3LJeS0jL7lTczl4unFmuf135cofbiIcMwqLHZqDEMbDab42GxWLBarfTr09sptf9YHDpynJgukVfeUIALn/OOj4/ngw8+oFevXld1Xy+//DKZmZm8+OKLV3U/Is2h/zsu0MhbRIALpx82btzIa6+9RlpamrPLEZHLUHiLCACrV6/m/vvv57bbbtONWURaOV1tLmICPj4+DX57WktKS0vTiFvEJDTyFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFqfw9fGmuKTU2WWIiIkUl5Ti6+Pt7DJaBYW3OEVwUADn8gsV4CLSKMUlpZzLLyQ4KMDZpbQKuj2qOIWnpwedOoZRUFjMufxCZ5cjIq2cr483nTqG4enp4exSWgWFtziNp6cH4WHtnV2GiIjpaNpcRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMm4O7sAEWmYYRgcyT7KyVOnqbHZCA4Oplu3GLw8PZ1dmog4mcJbpBXatmMn7y3+iILCQtq29cXd3Z2SkvO0adOGcWOT+OnkFNzd9ecr8mOlv36RyygsN0h47Bx5z4Y2q/2B0zamzC9i92MhjW7zwUefsGz5SkaOGMaEcWNoH3KhbXlFBV98sZWPPllG1qHDPPTA/Xh4eDSrLhExN53zFlP519ZKss/VOLuMOtbtt/Cz14pbpK8tW7ezbPlK7p5+B9Om3kpu7gkenv0o9z8wk48/+ZSk0SP445yHOXnqFG+89XaL7FNEzEfhLaby9tZKss/ZmtTGsF+lYv6j/088eGRC2+/cj81m473FHzI2aRSDB/bn1OnT/GP+a8R2i2FSyng+2/wFK1atITysA3ffdQdbtm4n++ix734AImI6Cm9xml05VsY+X0DcH85y/9slVFRfSNlf/V8Jsz8sBWD/KRvdfn+Wk8U1DJ6bT+ZJK796p4SFX1Rcto+FX1Rww0uFjHmugAfeKQEg/tFz/HNTBQOfzKfn4+f48KsqRy1bDlcz9C/5xP3xLHcuLKa0qvGJ/1mWhcc+LnW8Xn/AwuC5+Vz3/87xxn/qbIz9B7MoOX+elAnJABw5cpQuUT/hzmlTGZ88hqFDBpG5/yAAPRN60OnajmzbvrPR/YvID4fCW5yizGLnjteL+e3Ytux6LASrAfPWlwPwx0l+fLyrisNnbfxpaSn3J/kSHuDGltnt6B7ehhdvvYY7h/hctg+AQ2dqePYWf55L83cs25NnZe1Dwfzpp348/smFwM0vM7hzYTEvp1/D14+FUG6xs+Czxofuxc5X2vnFv0qYk9KWDb8LblLbEyfyCGnXjoBrrgFg6JBBzHnkIVxcXCguLmHPvkziYrs6to/uEsWJvJPNqlNEzE3hLU6x6aCFa4PcSEnwxKuNC3cP82FtpgWA9v6u/DrJlzteL+ZEUQ33jvBpch8Aw7p6cN21bWjj5uJY9uskX9p6upDcw5OCMoNyix1fTxeW/jqIhIg2eLi7MDrWk8NnmzY1X+vzw9VEt3dnUqIXgT6u3DGk4dobYrXa8PBoU2/5+dJS/vTnp4gID2NSynjHcg8PD6xWa7PqFBFz09Xm4hSnSgz2HsV98gAABl9JREFU5lmJ+8NZAOyAv9e37yWnDvLmLyvKmJPStk74NqWPhrj8pyu3/2xWY4Cvpwt782z8bvF5yi12CssNBkQ17yru/FKDiEC3ZrUNCgwgP7+Ampoa3Ny+7WPr1h1UWaq49+67cHH59mdx+sxZggIDm7UvETE3hbc4Rai/K4OjPfi/uxsOn/mbyomPcOe1zytIH+yNd5v6AX6lPhprT56Vp1aU8fH9QXS4xpUFn1XwVU7zRrQh/q4UlhvNatujexyW6moyvtpN/359HMu7x3UjPDyszsfCSkrOk7l/Pz+7/dZm7UtEzE3T5uIUw2I82HPCxsq9FmwGbDxYzcsbL5yvPl5Qw6ubK3gl/Rr6RLbhb2u+PY/t4+HC8cIabDWX76Mp8kvt2O1g2O0cOWfj491V1DQvfxka7UHmSRtf51ox7PDpN1VXbvQfgYEBDBk8kH+/9wElJecdy7ftyOD9D5c4Xtvtdha+9Tb+/v4MHNCveYWKiKkpvMUpAnxcWXhnAC+sLSPuD2d5akUZw2Iu3PbzDx+V8rNBPnQMdGN2Slve2FLhOAd920BvnlpRxoLN5ZftoylGdPNgeFcPRj1TwD1vljAwygNbTfM+X+bn5cKLt/lz379KGPqXfE4VN+1dwO1TbsbTy5Mnn36OnNwTAPzPDdfz+B9nA1BaVsaL/5jPvsz9/OLnM3SXNZEfKZeS0rKr/CnY75/dbq/33DAMx792ux3DMKix2agxDGw2m+NhsViwWq3069PbKbVL6xX/6Dkstrp/Lk/e6MfNfb1bdD+lpWW89MoCDmYdontcLF2iOuPu3oaTp06x++s9+Hh788v77qZL1E9adL8iYh4Kb4W3tFK7v/6GbTsyLvpikiASE+IZOmRwg1eli8iPh+bcRFqp6xJ7cl1iT2eXISKtkM55i4iImIzCW0RExGQU3iIiIiaj8BYRETEZXbAmTrM718YLa8pYm2nhB/eRBxET2nhf428q9H3y9fEmOCgAT8/m3bb4h0jhLU6xO9fGpL8VYAdcXcCu9BZxupgukc4uoUHFJaXknDhFp45hCvD/UHiLU7ywpgw7kNbPi7/8jz8e7g1/+YiIXH0RM884u4TLCrjGD4CCwmLCw9o7uZrWQee8xSnWZlpwdUHBLSKNEnCNH+UVlc4uo9XQyFucws6FqXIFt4hI02nkLSIiYjIKbxEREZNReIuIiJiMwltERMRkFN4iIiImo/AWERExGYW3iIiIySi8xTSioqKcXYKISKug8BYRETEZhbeIiIjJ6PaoIiLSZO+//z5Wq9Xx+uabb8bd3Z2qqio++ugjx3JPT09uvPFGAHbs2MHhw4cd6/r160d0dPT3V/QPiMJbRESazGKxUF1dXW+53W6nqqrh7wW32Wx11tXU1Fy1+n7oFN4iItIo58+fp6KiArgQ0hc7c+YMbm5uWCyWOssNw+D06dMAjra1SkpKHOvat2+Pq6vO5DaWwltERBplz549ZGVlNbhu9erVDS6vrq5mxYoVDa7bt28f+/btA+C2227D09OzZQr9EdDbHBERaXF+fn64u2t8eLUovEVEpMWNGDGCdu3aObuMHyyFt4iIiMloTkNERFqEm5tbvde1ywzDqHeRmzSfwltERFrE9ddf77jozNPTk5EjRzo+DrZ9+3ays7OdWd4PisJbRERaxMU3Z5k0aRI7d+50fBRMWpbOeYuIiJiMwltERMRkNG0uIiItbsWKFRiG4ewyfrAU3iIi0ijdunUjPDz8qvTdpk2bq9LvD5XCW0xDV6qKOFe7du1045VWQue8RURETEbhLSIiYjIKbxEREZNReIuIiJiMwltERMRkFN4iIiImo/AWp3ABXF2g2qZvGRIRaSqFtzjFmO6eGHaY9cF5BbiIXFFxSSm+Pt7OLqPV0E1axCkeGNuWtZkW3t1RxeKdVRjKbxG5hOKSUs7lF9KpY5izS2k1FN7iFNdd686y3wTzwpoy1mZanF2OiACHjhx3dgkN8vXxplPHMDw9PZxdSquh8Banue5ad964K8DZZYiImI7OeYuIiJiMwltERMRkFN4iIiImo/AWERExGYW3iIiIySi8RURETEbhLSIiYjIKbxEREZNReIuIiJiMwltERMRkFN4iIiImo/AWERExGYW3iIiIySi8RURETEbhLSIiYjIKbxEREZP5/6CzcMYfmXHcAAAAAElFTkSuQmCC)

- **Set external\_id from:**

  Use this option to select how you want to set the `external_id`, either from the Event Property you specify or directly from the Value you provide.

- **external\_id**

  Depending on the previous selection, here you can specify the value or the client event property (e.g. client\_id) that corresponds to the `external_id` user identifier for Braze API.

##### Braze User Alias Object

This configuration section is enabled if you have selected the `user_alias` as the **Braze User Identifier**.

![The Braze User Alias Object section with \&quot;Update existing users only\&quot; checked, and Event Property dropdowns for setting the User Alias Name and User Alias Label](/assets/images/user_alias_object-b7392e429e51269631693e8083b3e47a.png)

###### Update existing users only

When enabled(default), this option will only update existing users. Uncheck this box to allow creating alias-only users.

###### User Alias Name

- **Set user alias name from:**

  Using this section you can select how you want to set the name of the user alias object: from an Event Property or directly from a Value you provide.

- **Alias Name**

  Depending on the previous selection, here you can specify the value or the client event property that corresponds to the User Alias Name.

###### User Alias Label

- **Set user alias label from:**

  Using this section you can select how you want to set the label of the user alias object: from an Event Property or directly from a Value you provide.

- **Alias Label**

  Depending on the previous selection, here you can specify the value or the client event property that corresponds to the User Alias Label.

## Snowplow Event Mapping Options

This section includes the mapping rules that concern a Snowplow event as claimed by the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/):

### Snowplow Self Describing Event

![The Snowplow Self Describing Event section with \&quot;Include Self Describing event\&quot; enabled and the Self Describing Event Location dropdown open, showing \&quot;Nest under schema name\&quot; and \&quot;Merge to root level\&quot; as options](/assets/images/snowplow_event_mapping_options-cfd9b92901b525b1cee1dcb7a6a2b004.png)

#### Include Self Describing event

This option indicates if the Snowplow Self-Describing event data will be included in the event's properties object that will be sent to Braze. By default, this option is enabled.

#### Self Describing Event Location

This section is available only if the [Include Self Describing event](/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/braze-tag-configuration/#snowplow-self-describing-event) option is enabled. Using this drop-down menu you can indicate the location where Snowplow Self Describing event properties should be added under Braze event properties. The available options are:

- **Nest under schema name** (default): The schema name will be used as a key in Braze event properties with the self-describing data as its value.
- **Merge to root level**: The self-describing properties will be added directly as Braze event properties without nesting.

### Snowplow Event Context Rules

This section describes how the Braze Tag will use the context Entities attached to a Snowplow Event.

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in event object

Using this drop-down menu you can specify whether you want to Include `All` or `None` of the Snowplow context entities in Braze's `event_object`.

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- The Entity name to add/edit-mapping (required).¹
- The key you could like to map it to (optional: leaving the mapped key blank keeps the same name).
- Whether to add in `event_object` or `user_attributes_object` of the Braze event (default value is `event_object`).
- Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if `Include Snowplow Entities in event object` is set to `All`), you can specify the context entities you want to exclude from the Braze event. In its columns you can specify:

- The Entity name (required).¹
- Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

## Additional Event Mapping Options

If you wish to pick other properties from the Client event and map them onto the Braze event, these can be specified in this section.

### Event Property Rules

#### Include common event properties

Enabled by default, this option sets whether to automatically include the event properties from the [Common Event definition](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) in the properties of the Braze event.

#### Additional Event Property Mapping Rules

Specify the Property Key from the Client Event, and then the properties' object key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client. These keys will populate the Braze event's properties object.

#### Include common user properties

Enabled by default, this option sets whether to include the `user_data` properties from the common event definition in the Braze User Attributes object.

#### Additional User Property Mapping Rules

Using this table, you can additionally specify the Property Key from the Client Event, and then the User Attribute key you could like to map it to (or leave the mapped key blank to keep the same name). You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (note the array index 0) or pick non-Snowplow properties if using an alternative Client.

## Advanced Event Settings

![The Advanced Event Settings section showing empty input fields for Event Name Override and Event time property](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAbsAAADICAYAAABxqE/FAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d15fJTl3e/xT/aNTEISAgmEhJAAgQREwm4IKCISBAtSooD6KPB4bM9TT33KgVotPdalnmp92VN7jsW6tVoqFhSFAGUTDFvYCZCQBUMSwpJlQrZJZjl/8GRqyBAWEwI33/frNS+d+7rnun73PTDfue5lcKs0VzvsdjsAdrsdh8OB3W7HZrVis9uxWq3Oh8VioampieHD7kRERORW4d7ZBYiIiHQ0hZ2IiBiewk5ERAxPYSciIoansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ5nZxdwq1mzZg1btmzh4YcfZujQoW2uu3v3blasWMGCBQuIj4+/QRV+f1eqe/HixTT/xNx33X///UyYMOFGlNimnTt38s0331BRUUGXLl0YNmwY99xzDx4eHu02xocffkhNTQ1PP/10u/UpIh1HYXeNDh06BMCBAweuGHZGFhoaSnJycotlsbGxHT7u4sWLSUhI4LHHHnPZ/vXXX/Pll18SGxtLUlISpaWl/POf/8RsNjNr1qzrGuPkyZO8/fbbPPDAA6SkpABgMpnw9NRfH5FbxS33t/XkyZOcOHGCe++912X7hg0biI+PJyYmpt3HLioqoqKigrCwMHJzc6mvr8fPz6/dx7kVdO3alXvuuaezy2hl9+7dBAUFsXDhQtzdLx6lX7ZsGVlZWaSlpeHv798u4zz44IPt0o+I3Bi3XNht3LiRnJwcbDYbkydPbtG2du1aNm/ezKlTp3jiiSfafeyDBw/i4eHB9OnTeffddzly5AjDhw93ttfX1/Ppp5+Sk5NDaGgoUVFRzrbt27fzxRdfsHDhQuLi4gD4zW9+g5eXFz/96U+pqalh9erVHD9+HIB+/frxgx/8AH9/f8rKynjjjTdITU3l1KlTFBUV0aNHD2bPnk337t0BqKqqYtWqVeTl5eHr68vQoUOZPHmy89Dd0aNHycjIoKKigu7du5OWluacibVV97X64IMPyMnJ4Ve/+hVeXl6YzWZeeuklxo4dy/Tp02loaOCLL77gyJEjeHt7k5iYSFpaGl5eXmRmZrJq1SqmTZvGjh07MJvNxMfHk56eTnl5OW+++SYA2dnZLF68mFdffbXV+FarlcbGRhoaGpzB9uCDD1JZWemciV2uhnPnzrUaIy0tjdWrVwOwevVqjh07xsKFC3nrrbeora1lyZIlACxdupSYmBj8/f05dOgQ/v7+TJs2jcTERAAaGxtZsWIFR48eJTg4mDvuuIP169czb948kpKSqKqq4vPPPyc/Px9PT0+SkpKYOnUqXl5e1/1eiMi/3HIXqMybN4+YmBg2bdpERkaGc3lz0MXExDB37tx2H9fhcHDo0CHi4uLo168fJpOJAwcOtFjnq6++4siRIwwaNIgBAwY4D3kCzg+9nJwcAMrLyykvLycpKQmAv/71r2RnZ3PPPfcwfvx4Dh8+zOeff96i/23bttGjRw9GjRpFSUkJX331lbO2Dz/8kPz8fFJTUxk4cCBbt27ln//8JwAFBQV88MEHBAYGMmnSJOx2O3/+85+prq6+Yt2XY7VaqaysdD6a+0pKSsJqtZKXl9die5u3/8MPP+Tw4cOkpKRwxx13sHPnzhbvI8CmTZsYPHgwcXFxZGdns3PnTkJCQnjsscdwc3MjKiqKRx991GVdY8eOpb6+njfeeIPNmzdjNpsJCwsjPj4eb2/vNmtwNcbAgQO5//77ARg+fDj33XffZffJ0aNHqaurIzU1lcbGRj799FNsNhtw8VzvgQMH6NevH4mJiWzfvr3Fa//2t79x8uRJpk6dysiRI9mxY0er/SIi1++Wm9l5e3szf/58li1bxqZNm4CL/w7fli1biImJYf78+c4PtfZUWFiI2Wxm4sSJuLm5MXDgQHbv3k1tbS0BAQFYrVb27t1LbGwsjzzyCABdunThyy+/BCA4OJioqChycnJIS0trFQLjxo3D39+f6OhoAHJzcykoKGhRw9ixY3nggQeAi4dUi4uLgYuHdouLi1ucU7pw4QKFhYUAbN26lYCAAObMmYOnpyexsbG89dZbHD58mJEjR7ZZ9+WcPHmSV155xfm8e/fuPPvssyQkJODh4UFOTg4JCQnk5OQQEBBAnz59KC4uJi8vj7S0NEaPHg1ARUUFe/fudW4XwEMPPcTAgQNpamriF7/4BcXFxaSmpjJo0CDc3NwwmUwMHDjQZV0pKSl4e3uzbt061q5dS0ZGBnfeeSfTp0/H19f3ijW4GqNPnz4A9OjRw/n+uNKzZ09nWFqtVjZv3kx5eTlhYWHs3r2b6OhoZ0h369aN5cuXO19bWlpKVFQUw4cPd47v4+PT5nsgIlfvlgs7cB14HRl0cPEQJlz8UC8vL6dXr17s3LmTQ4cOMXr0aCorK7HZbM4PRgBfX98WfSQmJrJ27VrMZrPzkGFERARw8YN0/fr1fPLJJ9TW1tLU1NRqW757vsnX15fGxkYAzp49C0CvXr2c7d+9gKO0tJSamhqWLl3aor/mWdmV6nYlIiKCSZMmOZ83fzD7+fnRt29fcnNzsdvt5OXlkZSUhLu7O6WlpcDFmWTzrLSZxWJptZ1eXl54eHg4t/NqjRw5kuTkZI4ePUpmZiZ79+7FbDazcOHCK9bwfQLGz88PNzc34F/7sLGxkaqqKqxWK3379nWue+l7O3bsWDZu3Mhrr71GQkICQ4YMaTNYReTa3JJhBy0DD+jQoLPb7Rw+fBiAt99+u0XbwYMHGT16tPNiiLYub09KSmLt2rUcPXqU/Px858wC4L333qO+vp4ZM2YQFhbGihUrOH369FXV1/wB25bQ0FBmz57dYpnJZMLhcFyxblcCAgIYNGiQy7akpCQ+++wz9u/fT319vXP22mzy5Mmtrtxsj3NTtbW15OTkEBERQUREBElJSSQlJfHuu++Sk5OD2Wzu8BpcaX5/2nqf7rvvPvr378+BAwfIzs5m+/btpKWlkZqa2iE1idxubtmwg38FXvP/d5T8/HxqamoYPXq08+ISgD179pCTk0N1dTXBwcF4enq2CCir1dqin7CwMHr06MGmTZtobGx0nq+rqamhrKyMMWPGkJCQAOA813M1wsPDASgpKXHO0DIyMqitrWXmzJl0796d/Px8goODCQ4OBuDUqVOEhIRgs9muWPe1SkxMZOXKlWRkZODr6+u8V69Hjx7AxYtpmq+Wrauro76+3vll4Wo0B/SlrFYry5cvp1+/fjz55JPO5V26dAEuhs3V1uBqjMuNeyVBQUF4eXk5DytDy5ms2Wxm8+bNJCUl8eCDDzJt2jRef/11srKyFHYi7eSWDjvo2JBr1nwhyvjx4+natatzuZubG8ePH+fgwYOkpKSQnJzMrl272LBhA4GBgc4LRL4rKSmJDRs2YDKZnFc9+vv7ExAQwIEDBwgODqakpISioqKr3rbo6GiioqJYv3499fX11NXVkZmZ6fygvPvuu8nNzeWdd95hxIgRlJeXs3v3bp566in69OlzVXVfqqqqis2bN7dYFhMTQ58+fQgICCAmJoaCggLuuOMO56yxd+/exMXFsWvXLux2O926dWtxq8DVMJlMnDx5ko0bNzJhwoQWARUUFERycjJ79uxh2bJl9OnTh4qKCvbt20efPn0wmUyYTKYr1nDpGCaTCbj458Df37/V/YVX4u7uzvDhw8nMzOQvf/kLYWFh7Ny509keEBDAkSNHOHbsGBMnTqSpqQmz2XzZmbOIXLtb7mrMG81ms3HkyBF69OjRIujg4u0Bnp6ezvN5U6ZMYdCgQWzevJlt27YxePDgVv01z+YSExOdh7Xc3d159NFHMZlMbNy4EavVSmJiIo2NjdTW1l6xRjc3N+bNm0dsbCxbtmzhwIEDjBs3znnlYGxsLPPmzcPDw4N169aRm5tLWlqacxZ4NXVf6vz586xdu7bFIzc3t9V2Nv+32dy5cxk6dCiHDx9mw4YNhIaGMnPmzCuO16z5dpPt27e7nGnNnDmTtLQ0qqqq2LhxI7m5uYwYMaLF1ZtXquHSMUJCQhgzZgxnz57lyJEjV13rd02ZMoU77riDY8eOcfjwYeehXTc3Nzw9PVmwYAGhoaGsWrWK9evXk5iYyPTp069rLBFpza3SXO1o/uknu92Ow+HAbrdjs1qx2e1YrVbnw2Kx0NTUxPBhd3Zy2SK3lua/O82HVDds2MCGDRt45plniIyM7OTqRIzvlj+MKXIrWLduHdnZ2YwYMYKmpia2b99Or169nOcQRaRjKexEboBx48Zx4cIFtm7dioeHBwkJCUydOvWaLswRkeunw5giImJ4+lopIiKGp7ATERHDU9iJiIjhKexERMTwFHYiImJ4CjsRETE8hZ2IiBiewk5ERAxPYSciIoansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ7CTkREDE9hJyIihufZ2QWI3Kyqq6vJyc3DbK7G18+XPjHR9IyM6OyyROQ6KOxELlF94QKfLF/Bzl178PDwwGQKpK6unoaGBvrERDNvTjqxfWI6u0wRuQYKO7nlZWZmUlhYyJw5c753X2fPneM3//tNPDw8ePqp+QwdMhhPT08cDgeFhSf5x+erefk3r/PfFj7BsDuHtkP1InIjKOxcsNvtLF68uNXy9PR07rzzznYd69y5cxQUFDBy5MjL1jFt2jTuuusu5/JVq1bh6enJ1KlT27WWq1VUVMTq1aspLS0lKCiI8ePHM2LEiA4dc82aNfj7+zN+/PgOG8NqtfLm7/+IyRTIf/70P3DDjT+9+wGHjhwhvFs3Hpv7MM8+89/5698+5f8te48XnutOr56RHVaPiLQfhV0bnnvuOYKCgjp0jPPnz7Nr1y6XYdds/fr1DB48GJPJ1KG1XI2KigqWLVvG5MmTmTdvHqdPn+bvf/87vr6+DB48+Hv373A4cHNza/V81KhReHh4fO/+27J56zbKyyt49ddLCfD3551l75F7Io8fzvwB+w8e4ne/f5vXXn6ROemzKCw8yacrVvI/fvKjDq1JRNqHwu4affzxx/Tq1Ytx48YB8NFHHxEbG8vYsWMpKiris88+o7KykoSEBGbOnIm3tzeZmZnk5+djt9vJy8ujd+/ezJs3j/3795ORkYHFYuGVV15hyZIlLseMioriiy++YO7cuS7b8/Pz+eyzz6itraVPnz6kp6fj6+vLtm3bOHHiBDabjW+//Zbo6GgmT57Mp59+SkVFBQMHDiQ9PR13d3fsdjtfffUVe/bsoUuXLjzwwAMkJCS0GiszM5OBAwcyZswYAEwmE/fffz9ff/01gwcPvu79c+DAASwWCxEREYwfP5533nmH3r17c/r0aZYsWcLXX39NQEAA9957LwAbN25k+/bt+Pv707NnzxY1ZmVlsW7dOux2O2PGjOGee+65qvd25649jBk9kq5dgwEoKT3Nw7MfYsTwYQxPHsaPn/lPThUX0y8+jrT77+P//PEdamtrCQgIuKr+RaTz6NaDa5SUlMTRo0cBsNlsnDhxgsTERCwWC++//z733nsvzz//PDabjc2bNztfl5OTQ2pqKj//+c9paGhg//79jB49mvT0dCIjIy8bdABTpkwhPz+fnJycVm01NTW8//77zJkzh+eff57Gxka2bdvmbD916hRTpkxh8eLF1NTU8Ne//pXHHnuMRYsWcerUKY4fPw7A119/TUlJCUuWLGH27NksX76c+vr6VuOVlJQQGxvbYllMTAwlJSXfa/+cPXuWhx56iFmzZgFQW1tLUlISzz77bKsa8vLy+Oabb1i4cCELFiygvLzc2VZUVMSaNWtYsGABP/nJT9i7dy95eXmX3bffVVxSSnxcX+fzX73wc0YMHwbAtu2ZBHbpQq//Cta4uFjsdjuny85cVd8i0rkUdm146aWXWLRoEYsWLeJ3v/sdAAMGDKC0tJS6ujoKCwsJDw8nKCiI3NxcQkJCSExMxMvLi5SUFI4dO+bsKyEhgZiYGPz8/IiNjW3xAX0l/v7+TJ06lZUrV2K1Wlu0+fj48OMf/5iePXvi6elJ//79OXfunLM9Li6Onj170qVLF+Lj4xk4cCChoaGYTCaio6Oddezdu5cJEybg5+dHdHQ0PXv2pKCgoFUt9fX1rWYyAQEB2Gw2mpqarnv/xMfHExUV5TxUGRAQwLBhw/D29m5VQ3Z2NsnJyURERBAcHMywYcOcbfv27SM5OZnw8HBMJhPJycktxmmL1dqEt5dXq+VfrsngH5+v5j9+/BT+/n4AzroaGxuvqm8R6Vw6jNkGV+fsvLy8iI+P5/jx4xQXFzvPU5nNZkpKSnjhhRec6/r6+rrs183NDZvNdk21DBs2jKysLDZu3NiqntLSUlasWEFjY6PzUKYr7u4tv9u4ublht9uBi/eU/eUvf3GeL7Pb7S7Pwfn6+lJbW9tiWW1tLR4eHnh6euLm5va998+V1NTUtJpdNjObzeTm5rJz507g4jm/QYMGXVW/wcHBlJ1pPVP7au16pj8wpcWs78x/zehCQkKutXwR6QQKu+swePBgDh06xOnTp1mwYAEAgYGB9O3bl/nz53fYuDNmzOCtt94iKiqKyMiLVwGWlJSQkZHB008/TVBQENu2baOoqOia+w4MDGTWrFlER0e3uV5UVBT5+fktrr48efIkPXv2dAZlR++fwMDAVoHbzGQyMXHiRCZMmHDN/Q5OHMSOnbtJu/8+57Y4HA4en/cI/fvFt1j3m527CAsLpUf38GvfABG54XQY8zokJCRw4sQJfH196dq1K3DxMFxJSQnZ2dnY7XZyc3PZunXrFfvy9vampqbG5fmxS3Xr1o2UlJQW56BqampwOBw4HA7OnTvHwYMHnbO1azFkyBDWr19PXV0ddXV1rF69GrPZ3Gq90aNHc+zYMXbs2EFNTQ15eXlkZGSQkpLiXKc9948rgwYNYt++fdTV1WG1Wp3nCOFi0GZmZnLmzBmamprYvn07ubm5V9Xv5EkTKTtzltVfZTiX2R0O/vjOu5z8zheIvPwCNm3+mqlTJl9X/SJy42lm14aXXnqpxfMZM2YwatQovL29iYuLIyoqytnm7+/P448/zqpVq/jkk08IDw9n5syZVxwjOjqawMBAXn75ZV588cUrrn/33Xdz4MAB5/N+/frRr18/Xn/9dbp27Ur//v05f/78NWzlRRMmTGDNmjX89re/xW63M3z4cJe3OoSEhPDkk0+yevVqVq9eTVBQEJMmTWLIkCHOddpz/7jSt29fhg0bxhtvvIGfnx/du3dv0TZx4kTee+89amtriYuLu+p7I7t3D2fenHQ++Ohjmpoamf5AGp6enry/7I/Odfbu28+y9z7kjiFJpKaMva76ReTGc6s0VzuaZwJ2ux2Hw4HdbsdmtWKz27Farc6HxWKhqamJ4cPa98ZqkZvJ9m928NHHy/Hz82XokMGEhIRQV1/H0aPHKTpVzPhxdzH3kdkdft+fiLQfhZ2IC2ZzNVu+3sax47lUV1fj6+dHn5je3DVmNH1i2j6vKSI3H4WdiIgYni5QERERw1PYiYiI4SnsRETE8BR2IiJieLrP7jIslkbKK6qorbvyzd4icvsK8PcjNCQYH5/Wv+MqNw+FnQsWSyNFxafpFhZCZIR+Dkqks53I/5b4vjfnLR9V5gsUFZ+md68IBd5NTIcxXSivqKJbWAjBQYGdXYqI3OSCgwLpFhZCeUVVZ5cibVDYuVBbV6+gE5GrFhwUqFMeNzmFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8BR2IiJieAo7ERExPIWdiIgYnsJOREQMT2EnIiKGp3/1QEQMZ8WKFTQ1NTmfz5o1C09PTxoaGli5cqVzuY+PDzNmzABgz5495OXlOduGDx9OXFzcjStaOpTCTkQMx2Kx0NjY2Gq5w+GgoaHB5WusVmuLNpvN1mH1yY2nsBMRQ6iurqaurg64GGrfdebMGTw8PLBYLC2W2+12ysrKAJyvbWY2m51t4eHhuLvrrM+tTGEnIoZw+PBhcnNzXbatX7/e5fLGxkbWrl3rsi07O5vs7GwAHnnkEXx8fNqnUOkU+qoiIredwMBAPD31Xf92orATkdtOamoqYWFhnV2G3EAKOxERMTzN40XktuDh4dHqefMyu93e6qIWMRaFnYjcFqZNm+a8yMTHx4fx48c7by/YvXs3BQUFnVmedDCFnYjcFr57M/nUqVPJyspy3logxqdzdiIiYngKOxERMTwdxhSR287atWux2+2dXYbcQAo7ETGE/v37ExkZ2SF9e3l5dUi/cuMo7ETEEMLCwnSjuFyWztmJiIjhKexERMTwFHYiImJ4CjsRETE8hZ2IiBiewk5ERAxPYSciIoansHMhwN+PKvOFzi5DRG4RVeYLBPj7dXYZ0gaFnQuhIcGcO1+hwBORK6oyX+Dc+QpCQ4I7uxRpg35BxQUfH29694qgvKKKc+crOrscEQFO5H/b2SW4FODvR+9eEfj4eHd2KdIGhd1l+Ph4ExkR3tlliIhIO9BhTBERMTyFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8BR2IiJieAo7ERExPIWdiIgYnsJOREQMT2EnIiKGp7ATERHDU9iJiIjhKexERMTwFHYiImJ4CjsRETE8hZ2IiBiewk5ERAxPYSciIobn2dkFiHSW0tNlFJ48SX1dAyZTIP36xREcFNTZZYlIB1DYyW3n22+L+Ojj5eTlF+Dr44N/gD/V1Rew2WyMSB7Gw+kPKfREDMat0lztsNvtANjtdhwOB3a7HZvVis1ux2q1Oh8Wi4WmpiaGD7uzk8sWuT4HDh7iD/93GfF9Y5nxg2n0je2Dm5sbVquVg4eO8Olnq7BYLPzP/3yGHj26d3a5ItJOFHYu2O12Fi9e3Gp5eno6d97Zvtt+7tw5CgoKGDlypMv2Xbt2ERsbS7du3QD4wx/+QFpaGjExMe1ax83s0n1wvU6XlbH0xVcZM2oEj859mNLTZfz5/Y8oKT3NgP79ePzRR/Dx9uH1N3/PhZoaXvzlL/D29mqnrRCRzqQLVNrw3HPP8dprrzkf7R10AOfPn2fXrl2Xbd+1axfnz593Pp8+fTo9e/Zs9zquhcPhwOFw3LAxLt0H12vFPz6nR4/uzJuTjs1m483fv42HhwfpP5zJ6bIylv35A/z8fPmPH/071dXVbNy05XuPKSI3B52zu0Yff/wxvXr1Yty4cQB89NFHxMbGMnbsWIqKivjss8+orKwkISGBmTNn4u3tTWZmJvn5+djtdvLy8ujduzfz5s1j//79ZGRkYLFYeOWVV1iyZEmLsV599VWqqqr429/+xn333ceYMWNYvnw5Dz74IH379uWDDz7AZDJRUFBAZWUlo0aNIjo6ms8//xyr1crdd9/trLOmpoa///3vFBYWEhERwezZswkNDW21fUuXLmXIkCGcOHECm83GtGnTGDRokLNtwIABZGdn88wzzxAcHMyaNWvIysrC19eXlJQU7rrrLgAyMzM5cuQIVquViooKoqOjmT17Nt7e3gBkZWWxbt067HY7Y8aM4Z577nE5xp/+9KcW+8BisVBYWMgTTzwBQEZGBtXV1fzwhz9s831raGjgwMHDLJz/OO7u7pSdOYsbbvz7/H8jNDSELgEB/PGdd3E4HJhMJlLGjmHn7j3cP/ne6/2jIiI3Ec3srlFSUhJHjx4FwGazceLECRITE7FYLLz//vvce++9PP/889hsNjZv3ux8XU5ODqmpqfz85z+noaGB/fv3M3r0aNLT04mMjGwVdACLFy8mMjKS9PR0xowZ47KesrIyFixYwI9+9CN27NjB3r17efbZZ3niiSfIyMigrq4OgOXLlxMeHs4vf/lLkpKSWL58+WW3sWvXrvzsZz9j9uzZLF++nPr6emebn58fS5YsISQkhM2bN3Pq1CkWLVrE/Pnz2bZtG8eOHXOuW15ezrx581iyZAkWi8W5P4qKilizZg0LFizgJz/5CXv37iUvL8/lGJfug6SkJPLy8mhsbATg+PHjzjBuy+myM9hsNuLj+gIQGdGD1175X4SGhmC1WtmxazcD+sfj5uYGQFzfWEpKT3f4DFZEbgyFXRteeuklFi1axKJFi/jd734HwIABAygtLaWuro7CwkLCw8MJCgoiNzeXkJAQEhMT8fLyIiUlpcUHf0JCAjExMfj5+REbG0t5eXm71Dh06FBMJhMRERF0796d5ORk/Pz86N27NwEBAVRWVlJbW8uJEyeYNGkSnp6ejB07luLiYhoaGlz2OXDgQNzc3IiNjaV79+4UFBQ420aPHo2/vz9ubm7s27ePiRMnEhAQQLdu3UhJSSErK8u5bu/evQkMDMTDw4Nx48Y598e+fftITk4mPDwck8lEcnJyi3313TEuFRYWRmhoKDk5OVRXV3P+/Hn69et3xf3U1NQEgLeXd6u2N3//NqdOlfDUgiedy7x9vLHZbNgVdiKGoMOYbXjuuecIuuQSdC8vL+Lj4zl+/DjFxcUMHjwYALPZTElJCS+88IJzXV9fX5f9urm5YbPZ2r1ed3d3PDw8Wjx3OBxUV1fjcDj49a9/3WL9CxcuXLbGZl26dKGmpsZlW01NDV27dnU+79q1KwcOHHC5bmBgoLMfs9lMbm4uO3fuBC6en7ua2VmzpKQksrOzaWhoIC4uDi+vK19EEvJfdZadOUNcl1jn8qJTxRzJPsYvlvyMwMAuzuVlZWcIMpnwcNf3QREjUNhdh8GDB3Po0CFOnz7NggULgIsf5n379mX+/PmdXF1rgYGBeHl5sXTpUtyv8cO7qqqKLl26uGwLCgqiqqrKeZVkZWXlZdf9bpvJZGLixIlMmDDhmmpplpiYyDvvvIPFYrnqkAwLCyWiR3cyd+wiru+/wi44yMR/W/gkffrEOJc5HA527NxN4qCE66pPRG4++tp6ssdB5QAABd1JREFUHRISEjhx4gS+vr7OmU18fDwlJSVkZ2djt9vJzc1l69atV+zL29ubmpqaFufFLm0vLy//XjPBLl26EBUVRUZGBjabjbNnz7Jy5crLno/at28fVquVI0eOcP78eWJjY12ud+edd7Jx40bq6+uprKzkm2++YejQoc72wsJCzp49S0NDA1u2bCEh4WJ4DB48mMzMTM6cOUNTUxPbt28nNzf3svVfug8iIyPx9fXl2LFjzj6vRtqUyWzdtp3jOf8a61RxKX98513nYU6AjPX/5FRxCVMmT7rqvkXk5qaZXRteeumlFs9nzJjBqFGj8Pb2Ji4ujqioKGebv78/jz/+OKtWreKTTz4hPDycmTNnXnGM6OhoAgMDefnll3nxxRdbtY8YMYKVK1dis9lITU297m1JT0/nH//4B7/61a/w8/Nj8uTJLs+JwcXDk6+88srFy/LT0/Hz83O53rhx46itreW3v/0tDoeDcePGMWTIEGd7165dWbFiBWVlZcTHxztncn379mXixIm899571NbWEhcX1+ZtHa72QWJiIkVFRZedSbpy15hRHDlylDffept/e3wuI4cnM2jgAN5f9kcArFYrX65Zx+erv+KR9FlERkZcdd8icnPTTeXSwtKlS3n66acJDw//Xv1kZmZSWFjInDlz2qmyljZs2IC3t/c1fwGw2e18snwFGzdtIapXTwYNTMDf35/KykoOHDpMTU0Nj6T/kPHj7uqQukWkc2hmJ7cUu91OZWUlWVlZPPXUU9f8eg93d+Y+/EPG3TWGr7dnknsij/qGBgIDu5Aydgzjx91F167BHVC5iHQmhZ3cUoqKili2bBmTJk1qcSXoteod1Yu5D7d9I7qIGIcOY4qIiOHpakwRETE8hZ2IiBiewk5ERAxPF6hchsXSSHlFFbV1rm/2FhEBCPD3IzQkGB+f1r+7KjcPhZ0LFksjRcWn6RYWQmTE97vfTES+vxP53xLfN7qzy3CpynyBouLT9O4VocC7iekwpgvlFVV0CwshOCiws0sRkZtccFAg3cJCKK+o6uxSpA0KOxdq6+oVdCJy1YKDAnXK4yansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8PSvHoiI4axYsYKmpibn81mzZuHp6UlDQwMrV650Lvfx8WHGjBkA7Nmzh7y8PGfb8OHDiYuLu3FFS4dS2ImI4VgsFhobG1stdzgcNDQ0uHyN1Wpt0Waz2TqsPrnxFHYiYgjV1dXU1dUBF0Ptu86cOYOHhwcWi6XFcrvdTllZGYDztc3MZrOzLTw8HHd3nfW5lSnsRMQQDh8+TG5ursu29evXu1ze2NjI2rVrXbZlZ2eTnZ0NwCOPPIKPj0/7FCqdQl9VROS2ExgYiKenvuvfThR2InLbSU1NJSwsrLPLkBtIYSciIoanebyI3BY8PDxaPW9eZrfbW13UIsaisBOR28K0adOcF5n4+Pgwfvx45+0Fu3fvpqCgoDPLkw6msBOR28J3byafOnUqWVlZzlsLxPh0zk5ERAxPYSciIoanw5gicttZu3Ytdru9s8uQG0hhJyKG0L9/fyIjIzukby8vrw7pV24chZ2IGEJYWJhuFJfL0jk7ERExPIWdiIgYnsJOREQMT2EnIiKGp7ATERHDU9iJiIjhKexERMTwFHYuBPj7UWW+0NlliMgtosp8gQB/v84uQ9qgsHMhNCSYc+crFHgickVV5gucO19BaEhwZ5cibdAvqLjg4+NN714RlFdUce58RWeXIyLAifxvO7sElwL8/ejdKwIfH+/OLkXaoLC7DB8fbyIjwju7DBERaQc6jCkiIoansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8BR2IiJieAo7ERExPIWdiIgYnsJOREQMT2EnIiKG9/8BVCP5Oaebrs0AAAAASUVORK5CYII=)

This section offers additional configuration options on the Braze event object:

### Event Name Override

You can use this option to override the name of the Braze event object or leave it blank to inherit from common event properties, which is the default. Please note that the `name` property of the [Braze event object](https://www.braze.com/docs/api/objects_filters/event_object/#event-object), that this option populates, is a required property according to Braze API.

### Event time property

This option enables you to specify the client event property to populate the event time (in ISO-8601 format) or leave it empty to use the current time (default behavior).

## Logs Settings

Through the Logs Settings you can control the logging behavior of the Braze Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Braze GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{

    "Name": "Braze",    // the name of the tag

    "Type": "Message",  // the type of log (one of "Message", "Request", "Response")

    "TraceId": "xxx",   // the "trace-id" header if exists

    "EventName": "xxx"  // the name of the event the tag fired at

}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# Braze Tag for GTM Server Side
> Forward Snowplow events to Braze from GTM Server Side using the Braze Tag with Track Users API for personalization and marketing automation.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/

The Braze Tag for GTM SS allows events to be forwarded to [Braze](https://www.braze.com/). This Tag works best with the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/), but can also work with other GTM SS Clients such as GAv4.

## Template installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click "Search Gallery" in the Tag Templates section
2. Search for "Braze" and select the official "By Snowplow" tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking "Add"

### Manual Installation

1. Download [the template file](https://github.com/snowplow/snowplow-gtm-server-side-braze-tag/blob/main/template.tpl) `template.tpl` – Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select "Save Link As…"
2. Create a new Tag in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

## Braze Tag Setup

With the template installed, you can now add the Braze Tag to your GTM SS Container.

1. From the Tag tab, select “New”, then select the Braze Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to Braze.
3. Enter the required parameters and then optionally
4. [Configure the tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/braze-tag-configuration/) to customize your Braze Tag.
5. Click Save

---

# Configure HTTP Request Tag for GTM Server Side
> Configure JSON request body construction, entity mapping, custom headers, and post-processing for the HTTP Request Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/http-request-tag-for-gtm-ss/http-request-tag-configuration/

In the following short video a complete example configuration of the Snowplow GTM SS HTTP Request Tag is presented.



Scenario: The example assumes that we want to send a POST HTTP Request to an example custom destination endpoint, where we would like the body of the request to have the following structure:

```json
{

  "api-key": "myApiKey",

  "user_identifier": ...

  "event_data": {

    ...

  },

  "user_data": {

    ...

  }

}
```

where, for this example:

- Our endpoint expects the `api-key` inside the request body.

- As our `user_identifier` we want to map the value of the `client_id` from the client event.

- Inside `event_data` we want to include:

  - the common event data
  - the Self-Describing event data
  - the performance timing data from the Snowplow [Performance Timing Context](https://github.com/snowplow/iglu-central/blob/master/schemas/org.w3/PerformanceTiming/jsonschema/1-0-0)), with `performance_timing` as the property name
  - the page view id from the Snowplow [web\_page context](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0), with `page_view_id` as the property name.

- Under `user_data` we want to map the `user_data` from the client event.

You can read on below for more details on each configuration option.

## Configuration options

### Destination URL (required)

Set this to the URL of your custom HTTP endpoint.

### Wrap the request body inside an array

By default, the JSON body is an object. For example:

```json
{ "myProperty": "myValue" }
```

This option allows you to wrap the resulting object of the request body inside an array:

```json
[{ "myProperty": "myValue" }]
```

### Include all event data in the request body

This option allows you to relay the full client event into the body of the request. Enabling this option, consequently disables both the Snowplow and the Additional Event Mapping Options, which allow to cherry-pick event properties and customize the request body.

### Use alternative separator to dot notation

Enable this option to use an alternative separator to dot notation when specifying possibly nested object paths. This setting **applies everywhere dot notation can be used** and it is useful when you want to allow dots or special characters in key names. Enabling this option reveals the text-box where you can specify the character you wish to use to denote nested paths.

#### Example

Let's imagine this property name: `user_data.address.city`

This is by default being interpreted as a nested path, where the dot character denotes a change in nesting level:

```json
  "user_data": {

      "address": {

          "city": "Foobar City"

      }

  }
```

If you wish it to denote a different nesting path where a key name may include a dot, for example:

```json
  "user_data.address": {

      "city": "Foobar City"

  }
```

then you can use this setting to set a different separator for nesting. As an example, if you set:

![The \&quot;Use alternative separator to dot notation\&quot; option enabled, with the Separator field set to a tilde (\~)](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZwAAAByCAYAAABneGd/AAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABpySURBVHic7d15WFNn2gbwmzUSFlGhClRpwqJUFAvFKqIWRcUNW5e6daHiYGtbdUangFu134BLhbbTOlpq3abuiK2Oa91aa11QBwVEEawgagSBsIQlJHm+PxxOjUQIVRLE53dduTTvOec9T96ccJ+8ORATeWkZ4X+I7v9Xo9EI/xIRNBoN1CoV1BoNVCqVcKuurkZNTQ38/XzBGGOM1cfU2AUwxhh7NnDgMMYYMwgOHMYYYwbBgcMYY8wgOHAYY4wZhLmxC2isbclViNpZCqWKGl5ZB0tzEywdY4fx/q2ecGWMMcbq89S9w3mcsAEApYoQtbP0CVbEGGNMH09d4DxO2DzJPhhjjDXOUxc4jDHGnk7PROAEupkYuwTGGHvmtfjACXQzwbQ+ZrgQafGn+6ioqIBUKsXp06e12rOzsyGVSpGdnf24Zert8uXLkEqlKC4uNtg+GyM7OxsBAQG4e/eusUthj9C3b19s27bN2GU8EcY+3m7fkeHkqdM4fOQ4ziafh7ykxCh1PC1adODUhk1XJxPsTtXAya7lvdPZs2cP/Pz8ms3+O3TogPDwcLRt29ZoNRnT/PnzMX369CfWX0FBAaRSKTIzM59Yn43R1MdXY8eruRxvOTm5+MeSzzB3wWL8+/ut2HfwEBK+W4+/zonGqm++4+B5hKfusmh9PRw23/yqwZ1SvljgQWq1GmZmZk+0T2tra4SHhz/RPpuLphivB2k0GpiatuhzwCfOGMdbysVLWLl6DTzcpJgf/Xe4SSUwMTGBSqXCxUtp2LHzByz+v6WInDMLHTq0N2htzd1Tf3S7OdR912KssLl8+TLGjRuHrl27IigoCImJiVrL165di4CAAPj6+mLWrFmQy+U6+ykvL8fcuXPh6+uLHj16YObMmSgtrXsp97Rp0zBz5kwUFxdDKpXi6NGjAIDc3Fy8/fbb8Pb2xuDBg3HgwAFhm9DQUMyZMwdDhw7F+PHjhenCbdu2YejQoejatSveeecdFBUVCdscO3YMI0aMQJcuXRAcHCz0p2v/D075JSUloWfPnsJfHweA6dOnY968eQCAyspKzJ07Fz4+PujTpw++/PJLqNVqnWPyn//8B8HBwejatStGjx6NlJQUYVl9/cTGxiIsLAwffvghevTogYCAACQlJek11rVjExMTg4CAAMTFxdW7vo+PDzZv3owDBw5AKpWioqICAHD8+HEMGzYMXbp0QUhICI4fPy7sPzY2FuPHj8eUKVPg5eWF6upqYdm6devwyiuvAABCQkIQGxsr1BUdHY0ePXrAx8cHkZGRwr502bBhA/z9/dGzZ08sW7aszhg/qr5HHV8Pio2NxXvvvYdFixahe/fu6N27t9ZxX1+tjxqvP3u81UpISECfPn3g5eWFt956C7///ruwLDQ0FHFxcZgyZQpefPFFhISE4NKlS48cu4fdkcmwKmEtAgN64e+zZ8LKygr/WPIZ3vvwr/h61bdwc5Pgk/lRaNeuLT7/6l9QKmv07vuZIC8to9pbcUkpFZeUUmGxnAqL5VRQWET59wpJll9At27fody8W3T9Rg5lZmXT5StX6b8XL9HZc+fJkJxny4SbLL+Qvj9RQH2X3BXa3vhXPh25eI9k+YWUcKSA/Bbf1dqm9tYYCoWCJBIJnTp1Sqs9KyuLJBIJZWVlERFRQEAAffrpp5Sbm0vbt28nDw8PSklJISKijRs3Ut++fen06dOUmZlJEydOpBkzZujc38yZM2nMmDF09epVyszMpJEjR9L8+fOJiCg9PZ0kEgkVFRVRVVUVJSYmkq+vLykUClKpVKRQKCgwMJCWLFlCN27coMTERPL09KTMzEwiIho5ciT5+/vT4cOHKScnR3hsQUFB9Ntvv9HZs2cpMDCQYmNjiYjo2rVr5O7uTlu3bqXbt2/Txo0bydPTk27fvq1z/w/WV1ZWRp07d6Zz584REZFSqSRvb2/69ddfiYho+vTp9Pbbb9PVq1fp1KlT1Lt3b9q0aVOd8cjOziYPDw/64YcfKCcnhxYuXEi+vr5UVVXVYD8xMTHk5eVFx44dI4VCQevXryc3Nze6ceNGg2NdOzZjx46l8+fPU35+foPrR0ZG0rRp00ihUBARUUpKCnl4eNC3335L2dnZtGrVKvLw8KBr164J9bm5udGGDRsoIyODNBqN8LhramooJyeHJBIJXbx4kZRKJRERTZ06lUaMGEEpKSl04cIFCgkJoVmzZuk8ls6ePUtubm6UkJBAGRkZFBsbSxKJhLZu3dpgfbqe34fV1r98+XK6cuUKLV++nDp37kxFRUUN1qprvB7neCMiWrNmDb300kt06NAhyszMpBkzZlD//v2FsRs5ciR5eXnRDz/8QOnp6RQWFkahoaE6x06Xf65cTQs/jSW1Wk01NTU0J2o+xSxdQcd+PkEfz11In8V/SUREJSUl9N6Hs2jf/kN69/0seGoDZ8QXd2lP8j2t0NE3bJoicBQKBUmlUjp+/LiwfNeuXXT16lUiIurbty/t3r1bWHbp0iVyd3enmpqaOvvLzMykgoIC4f53331HQ4cOJSKq8wLbvXs3+fr6Cuvu2LGDgoODtfqbMmUKxcfHE9H9F9yXX35Z57E9WPfy5ctp8uTJREQkl8spLS1Nq79u3brRwYMHde7/4fqmTp1KS5YsISKi48ePk6+vL6lUKrp58yZJpVK6e/eusG1CQgK98cYbdcbj6NGj5OnpSaWlpULNGzZsILlc3mA/MTEx9M4772j1N2rUKIqLi2twrGvH5syZM8Ly+tYnIpo3bx69//77wv2ZM2fStGnTtPYfHh5OH3/8sVDfpEmT6jzmWvn5+SSRSITj6Pr16ySRSOjy5cvCOqmpqSSVSikvL6/O9jNnzqSpU6dqtfXq1UsInIbqe/j5fVhMTAy9/vrrwv2qqiqSSqWUnJysV60Pj9fjHG8ajYZ69epFa9eu1arHz8+PkpKSiOj+8b9s2TJh+c8//0weHh6kVqsf+RhrVVZW0pSID+j02WQiIrp1+w79PWoB3btXSEREyecu0JSID4SThk1bttPCxTEN9vsseWo/w7lwk/DNSTUAMwR3vj8z6GRnYrTPbMRiMSIiIvD+++/j1VdfxYABAzBs2DCIxWKUlpYiLy8PkZGRiI6OBnD/y+7UajXu3LmDjh07avXl6OiI+Ph4/PLLLygqKoJKpYKzs7NedWRkZODGjRvw9vYW2pRKJdq0aSPct7S0rLOdnZ2d8H8rKytheqN169Y4d+4cFi1ahOvXr0OlUkGhUGhN/dRn+PDh+Oc//4moqCgcOXIEgwcPhpmZGS5fvgwiwoABA4R1VSoVHBwc6vQREBAAPz8/BAUFYciQIQgODsabb74JU1NTnDlzRu9+anl5eSE3NxeAfmNtYfHHFY6NfW4yMjIwduxYrTZ/f3+tac4H+29IRkYGxGIxvLy8hDZvb2+0atUK165dg4uLi9b6N27cQHBwsFabufkfL3t96mvIg8eOSCSCqakpFAoF8vPzG1Ur8HjHW3FxMe7evYuXX35Zq57u3btrXXTxYL1isRgqlQo1NTUQiUT19n9HdhdqtRoe7m4AAGenDli+5FMA94+5U2fOoktnD5iY3J/md3eT4ujxX0BEQtuz7qkNHEB36DRF2FhYWMDU1BQ1NdrzsUqlEgCEAzUyMhITJkzA4cOHsWnTJqxYsQLbt2+Hvb09ACA+Pl7rxQcATk5OdfYXFRUFlUqFLVu2wMnJCRs3bsTGjRv1rtfPzw/Lli3TarO2ttZ7+welpaVhxowZiI+Px4ABA2BhYdGoq5YGDRqE6OhoXLt2DYcPH8bSpUuFZZaWlti7d6/W+ro+lBeJRNi8eTMuXLiAY8eOYcGCBXBxccGmTZsa1U+tB5/Hxo51Y9e3tLTU+gEP3L84oPbYaSxLS8s6FxbQ/76VV1efJiYmOk8wmqq+x6kVeLzjrfZxPvzcq9XqJ/J4ao8bS4u64/nFV/9Cfv49LJj78R/1iCyhVquhIYIZBw6AFnDRQG3oJOdQk72zsbCwQMeOHZGWlqbVnpaWBisrKzg7OyMrKwtxcXHo1KkTwsPDkZSUhNatW2P37t2ws7ODo6Mj8vLy4OrqKtwcHBzqvNgB4OTJkxg/frwQRvV9IPzwmZObmxuysrLQvn17YT+Ojo71nvHX58yZM5BIJBgyZAgsLCygUqm0XrwNnblZW1ujX79++Pzzz1FVVYWAgAAAgLu7O5RKJcrKyoQ6n3/+eZ11Hjt2DNu3b4evry9mz56NH3/8EcnJyUhJSdGrnwcvWgCA1NRUuLq6AmjcWOuz/sPj4eHhgQsXLmi1nTt3Dl26dKl3P/X1V15ejqysLKEtLS0N1dXVOvuUSCS4evWqVhvRH6+Phup7nDNzfWp9uP/HOd5sbGzQoUMH/Pe//xXaqqurkZaWpvd416ft/2YJZA/9zk/uzTykpWcgYmoYbG1thHaZ7C5a29nBjK88FLSIkagNnaacRps6dSpWrVqFbdu24cqVK9izZw+WL1+O8PBwmJqawsbGBmvWrMEXX3yBmzdv4uTJk5DJZJBKpQCAiIgIfPXVV9izZw9u3ryJlStXYuzYsVov/loSiQT//ve/kZKSgh07dmD16tWPPENr06YN5HI5Tp48Cblcjtdeew0WFhb429/+hszMTKSkpOD111/H/v37/9TjfuGFF5CZmYmkpCScP38eH330ERQKhVDPw/vXZfjw4Thw4AAGDRokBKxUKsXAgQMxe/ZsJCcnIzs7GzNmzMCSJUvqbK9UKvHJJ59g7969uHXrFvbu3Qtzc3N07NhRr35Onz6NxMRE5ObmIi4uDtevX8eYMWMaPdb6rG9vb4+MjAykpqZCpVLhL3/5Cw4ePIh169bh999/R0JCAk6cOIF3331Xr/G3s7ODqakpjh49CplMBldXV4SEhGDOnDm4dOkSLl68iKioKAwePBidOnWqs/2kSZOwb98+HDlyBJWVldi0aRNu374tLG+oPn2e30fRp9aHx+txj7f33nsPn3/+OY4cOYKsrCxERUVBJBJh+PDhjapdFweHdnDq0B6/nTqj1W7f2g7vR4RDInlBaCMinDp9Ft5dtWc0nnUtInCA+6HTlJ/ZTJo0CVFRUVi7di1GjRqF+Ph4hIeHY9asWQDu/wLamjVrcPToUQwaNAiRkZGIiIjAsGHDAADvvvsuwsPDERMTg0GDBuH48eOIi4vTecb22WefoaysDG+++SZ27dqFWbNmQSwW6/xB+Morr6B3796IiIjA+fPnIRaLsX79esjlcoSGhmLatGkYMmQIBg8e/Kce98CBAxEREYGYmBh89NFH6N69O4KCglBWVqZz/7oEBwejVatWGDp0qFb7ihUr0LVrV4SHh+O1116Dubm5MJ4PGjJkCD7++GMsW7YMAwcOxObNm7Fy5Uq0b99er366d++OU6dOYdSoUdi5c6fwTrSxY63P+mPGjIGJiQkmT56MsrIyeHl5YeXKldiyZQtCQkLw448/4ttvv9X6jK0+lpaWmDZtGr7++mts2LABALBs2TK4u7tj8uTJCAsLg4+PD+Lj43Vu//LLLyMyMhKRkZEIDAxEWloa3N3dheUN1afP81ufhmp9eLwe93h76623EBYWhnnz5iE0NBRFRUXYsmULxGJxo2vXZfiwEPx84ldcufrHZ0I3825jVcJ3WlO1Bw4dxs28WxgW8udedy2Viby0TPgpXXu2XTsFodFohDlXtUoFtUYDlUol3Kqrq1FTUwN/P1+DFSyJyn/sv/ZsaW6C35c+94QqYs1ZbGwsMjMzsX79emOXwlqI1QlrkXLxEt4NexOv+L+stUylUuE/+w7ixz17MWnCOAwaGGSkKpunp+6igaVj7J7IF7Axxtif8ZepYdiyLRGrE9Zi776D6PqiF8RiMYqLi5FyKRXl5eV4561JeLVfoLFLbXaeusAZ79+Kv62TMWY0ZqameHPiG+gXGIBffv0NmdeyUFlVBVtbG/TtE4BX+wWiTRt7Y5fZLD11U2qMMcaeTi3mogHGGGPNGwcOY4wxg+DAYYwxZhAcOIwxxgyCA4cxxphBcOAwxhgzCA4cxhhjBsGBwxhjzCA4cBhjjBkEBw5jjDGD4MBhjDFmEBw4jDHGDIIDhzHGmEFw4DDGGDOIp+77cIwlOzsb+/btg0wmg62tLfr164eAgABjlwUAOHPmDKRSKRwdHY1dCmOMPRIHjh4qKiqwfv16jB07Fi+++CLy8/OxYcMG2Nraolu3bk26byKCiYlJveucOXMGdnZ2jQqc2u8+aqhvxhh7Ujhw9FBYWAhTU1P4+PgAAFxcXBASEgKFQgHg/hfV7d27F8nJybCxscHIkSPh5eUFmUyGtWvXwtXVFTk5ObC1tcWECROEYMjOzsbOnTuhUCggkUgwYcIEtGrVCjKZDAkJCejUqRPu3LmD6OjoR667dOlSyOVybN26FUOGDEFAQABycnKwa9cuFBYWwtXVFWPHjoW9/f1vIFy0aBG6dOmC9PR0zJo1C+3atTPOoDLGnjn8GY4enJycYGNjg8TERBQVFQEAfH190atXLwDAL7/8glu3biE6Ohrjx4/Htm3bUFlZCQAoKytDnz59MHfuXHh5eWHHjh0AgPLycqxfvx6TJ0/GggULoFQqceLECWGfCoUC3bp1w+zZs+tdNyoqCs7OzpgwYQICAgJQWVmJdevWISgoCJ988glcXV2xceNG4R0NAFhZWSE6Ohpt27Y1yPgxxhjAgaMXc3NzfPDBB2jVqhW+/vprrFy5EleuXBGWnz9/HkFBQbCysoKrqytcXFxw/fp1APd/uL/wwgsAgP79+yMnJwdVVVUQiUT48MMP4eLiAnNzc3Tu3BkFBQVCn9bW1vDz84OlpWWD6z7oypUrcHR0hI+PD8zNzTFw4EAUFxcjPz9fWKd3794Qi8U8ncYYMyieUtOTWCzGiBEjMGzYMKSnp2P79u0YPXo0vL29UVpaiu+//174Aa7RaNC9e/c601UWFhZo1aoVysvL4eDggNu3byMxMRFKpVKYKtPFwsJC73XLy8vRpk0b4b6pqSlat24NuVyO9u3bP6HRYIyxxuPA0cOlS5dQXFyM/v37w9TUFN26dcOtW7eQnp4Ob29v2NraYty4cXB1ddXaTiaTad2vqqpCVVUVbGxscOvWLRw4cADTp09H69atceLECeTm5urcf2PWtbe3R2pqqnBfo9GgpKQENjY2jzkKjDH2eHhKTQ9t2rTBkSNHcPnyZdTU1KCoqAhXr16Fk5MTAMDHxweHDh1CRUUFKioqsGfPHpSUlAC4f4Vbeno61Go1fvrpJ7i6ugrvcogIRISCggJcvHgRGo1G5/4bWtfS0hKFhYVQq9Xw9PTEvXv3kJaWBrVajZ9//hk2NjZwdnZu+oFijLF68DscPXTs2BETJ07ETz/9hO+//x7W1tZ46aWXEBgYCAAICgrCvn37sGLFCmg0Gvj7+8POzg6VlZUQiURITU3F9u3b0a5dO0ycOBEA4OnpCU9PT8TFxaFNmzbo3Lkz7t27p3P/Da3bs2dP7Nq1C2q1Gv3798eUKVOQlJSEbdu2wcXFBWFhYfx5DWPM6EzkpWXC5Uu1VzLVnj1rNBoQETQaDdQqFdQaDVQqlXCrrq5GTU0N/P18jVN9M1d7efPChQuNXQpjjBkdT6kxxhgzCA4cxhhjBsFTaowxxgyC3+EwxhgzCA4cxhhjBsGBwxhjzCD493D0VF2tRGGRHIqKSmOXwhhrxqzFVmjX1h4ikaWxS2l2OHD0UF2tRG7eHTg6tIWz03PGLoexZ9617Bx4uLk2vKIRyEvKkJt3B52ed+LQeQhPqemhsEgOR4e2sG9ta+xSGGPNnH1rWzg6tEVhkdzYpTQ7HDh6UFRUctgwxvRm39qWp9914MBhjDFmEBw4jDHGDIIDhzHGmEFw4DDGGDMIDhzGGGMGwYHDGGPMIDhwGGOMGQQHDmOMMYPgwGGMMWYQHDiMMcYMggOHMcaYQfBfi2aMtXiJiYmoqakR7o8bNw7m5uaoqqrCrl27hHaRSITRo0cDAJKTk5GVlSUs8/f3h7u7u+GKboE4cBhjLV51dTWUSmWddiJCVVWVzm1UKpXWMrVa3WT1PSs4cBhjLVJpaSkqKioA3A+WB929exdmZmaorq7WatdoNJDJZAAgbFurpKREWPbcc8/B1JQ/kWgsDhzGWIuUmpqKzMxMncsOHTqks12pVGL//v06l6WnpyM9PR0AMGnSJIhEoidT6DOEI7oJpaenY/DgwfDz88OWLVuE9m+++QaLFy82YmWMsQfZ2trC3JzPv5saB04TioyMxKBBg7B48WIsXboUq1evRmlpKfbt2wdfX19jl8cY+5/+/fvDwcHB2GW0eBzpTaSiogJ5eXn44IMPIBaL0bFjR0RERGD58uXo168fhg0bZuwSGWPMoDhwmohYLEZKSopw38fHB7/99htkMhlcXFyMWBljDADMzMzq3K9t02g0dS40YI+PA8eAzMzMOGwYayZCQ0OFD/5FIhFeffVV4dLns2fP4vr168Ysr0XiwGGMPZMe/IXPESNG4Ny5c8Jlz6xp8EUDjDHGDIIDhzHGmEHwlBpj7Jm3f/9+aDQaY5fR4nHgMMZapM6dO8PZ2blJ+rawsGiSfls6DhzGWIvk4ODAv8zZzPBnOIwxxgyCA4cxxphBcOAwxhgzCA4cxhhjBsGBwxhjzCA4cBhjjBkEBw5jjDGD4MDRg7XYCvKSMmOXwRh7SshLymAttjJ2Gc0OB44e2rW1R8G9Ig4dxliD5CVlKLhXhHZt7Y1dSrPDf2lADyKRJTo974TCIjkK7hUZuxzGGIBr2TnGLkEna7EVOj3vBJHI0tilNDscOHoSiSzh7PScsctgjLGnFk+pMcYYMwgOHMYYYwbBgcMYY8wgOHAYY4wZBAcOY4wxg+DAYYwxZhAcOIwxxgyCA4cxxphBcOAwxhgzCA4cxhhjBsGBwxhjzCA4cBhjjBkEBw5jjDGD4MBhjDFmEBw4jDHGDIIDhzHGmEFw4DDGGDOI/wcoyo0tIkmfFQAAAABJRU5ErkJggg==)

then the tilde character (`~`) denotes nesting everywhere "dot notation" can be used. You can now denote the example path above as `user_data.address~city`.

It is also possible to use more than one character as alternative separator, for example:

![The \&quot;Use alternative separator to dot notation\&quot; option enabled, with the Separator field set to two dots (..)](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZsAAABxCAYAAAADMA6oAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABouSURBVHic7d15WFNn2gbwGwTCLipUgSombFJRLBSriFoUFTdsXerWhYqDrW3VGZ0CbnX6DbhUaDutY4daResuYqt1rVtrrQvqoIAoghXcEATCEpaQ5Pn+cDglEiWop0F8fteVS/Kec97z5M0J98mbgzEqLa8gIkKdup81Go3wLxFBo9FArVJBrdFApVIJt5qaGtTW1sLfzxeMMcaYLsaGLoAxxljLx2HDGGNMdBw2jDHGRMdhwxhjTHQcNowxxkTHYcMYY0x0JoYu4FFsSalG1PYyKFXU+Mo6mJkYYckYW4z3N3/ClTHGGNPlqXxn8zhBAwBKFSFqe9kTrIgxxtjDPJVh8zhB8yT7YIwxpp+nMmwYY4w9XZ6ZsAl0NTJ0CYwx9sx6JsIm0NUI0/q0wrlI00fuo7KyEjKZDCdPntRqz8nJgUwmQ05OzuOWqbeLFy9CJpOhpKTkT9tnU+Tk5CAgIAB37twxdCnsAfr27YstW7YYuownwtDH263b+Th+4iQOHjqK0ylnIS8tNUgdzV2LD5u6oOnqaISdaRo42ra8dzi7du2Cn59fs9l/hw4dEB4ejrZt2xqsJkOaP38+pk+f/sT6KywshEwmQ1ZW1hPrsynEPr6aOl7N5XjLzc3DPxd/irkL/oHv1m/Gnv0HkPBtIv46Jxor//Mth859nspLn/V1f9D851cNbpfxhQH1qdVqtGrV6on2aWVlhfDw8CfaZ3MhxnjVp9FoYGzc4s8BnyhDHG+p5y9gxder4O4qw/zov8NVJoWRkRFUKhXOX0jHtu3f4x//twSRc2ahQ4f2f2ptzVWLOKpd7Ru+WzFU0Fy8eBHjxo1D165dERQUhKSkJK3lq1evRkBAAHx9fTFr1izI5XKd/VRUVGDu3Lnw9fVFjx49MHPmTJSVNbxce9q0aZg5cyZKSkogk8lw+PBhAEBeXh7eeusteHt7Y/Dgwdi3b5+wTWhoKObMmYOhQ4di/PjxwhThli1bMHToUHTt2hVvv/02iouLhW2OHDmCESNGoEuXLggODhb607X/+tN8ycnJ6Nmzp/CVFQAwffp0zJs3DwBQVVWFuXPnwsfHB3369MEXX3wBtVqtc0x+/PFHBAcHo2vXrhg9ejRSU1OFZQ/rJzY2FmFhYfjggw/Qo0cPBAQEIDk5Wa+xrhubmJgYBAQEIC4u7qHr+/j4YOPGjdi3bx9kMhkqKysBAEePHsWwYcPQpUsXhISE4OjRo8L+Y2NjMX78eEyZMgVeXl6oqakRlq1ZswYvv/wyACAkJASxsbFCXdHR0ejRowd8fHwQGRkp7EuXtWvXwt/fHz179sTSpUsbjPGD6nvQ8VVfbGws3n33XSxatAjdu3dH7969tY77h9X6oPF61OOtTkJCAvr06QMvLy+8+eab+P3334VloaGhiIuLw5QpU/DCCy8gJCQEFy5ceODY3e92fj5WJqxGYEAv/H32TFhYWOCfiz/Fux/8FV+t/AaurlJ8PD8K7dq1xWdf/htKZa3efbdopeUVJC8rF24lpWVUUlpGRSVyKiqRU2FRMRXcLaL8gkK6ees25d24SVev5VJWdg5dvHSZ/nv+Ap0+c5b+TE6z84VbfkERrT9WSH0X3xHaXv93AR06f5fyC4oo4VAh+f3jjtY2dbemUCgUJJVK6cSJE1rt2dnZJJVKKTs7m4iIAgIC6JNPPqG8vDzaunUrubu7U2pqKhERrVu3jvr27UsnT56krKwsmjhxIs2YMUPn/mbOnEljxoyhy5cvU1ZWFo0cOZLmz59PREQZGRkklUqpuLiYqqurKSkpiXx9fUmhUJBKpSKFQkGBgYG0ePFiunbtGiUlJZGHhwdlZWUREdHIkSPJ39+fDh48SLm5ucJjCwoKot9++41Onz5NgYGBFBsbS0REV65cITc3N9q8eTPdunWL1q1bRx4eHnTr1i2d+69fX3l5OXl6etKZM2eIiEipVJK3tzf9+uuvREQ0ffp0euutt+jy5ct04sQJ6t27N23YsKHBeOTk5JC7uzt9//33lJubSwsXLiRfX1+qrq5utJ+YmBjy8vKiI0eOkEKhoMTERHJ1daVr1641OtZ1YzN27Fg6e/YsFRQUNLp+ZGQkTZs2jRQKBRERpaamkru7O33zzTeUk5NDK1euJHd3d7py5YpQn6urK61du5YyMzNJo9EIj7u2tpZyc3NJKpXS+fPnSalUEhHR1KlTacSIEZSamkrnzp2jkJAQmjVrls5j6fTp0+Tq6koJCQmUmZlJsbGxJJVKafPmzY3Wp+v5vV9d/cuWLaNLly7RsmXLyNPTk4qLixutVdd4Pc7xRkS0atUqevHFF+nAgQOUlZVFM2bMoP79+wtjN3LkSPLy8qLvv/+eMjIyKCwsjEJDQ3WOnS7/WvE1LfwkltRqNdXW1tKcqPkUs2Q5Hfn5GH00dyF9Gv8FERGVlpbSux/Moj17D+jdd0v2VIfNiM/v0K6Uu1qBo2/QiBE2CoWCZDIZHT16VFi+Y8cOunz5MhER9e3bl3bu3Cksu3DhArm5uVFtbW2D/WVlZVFhYaFw/9tvv6WhQ4cSETV4ce3cuZN8fX2Fdbdt20bBwcFa/U2ZMoXi4+OJ6N6L7Ysvvmjw2OrXvWzZMpo8eTIREcnlckpPT9fqr1u3brR//36d+7+/vqlTp9LixYuJiOjo0aPk6+tLKpWKrl+/TjKZjO7cuSNsm5CQQK+//nqD8Th8+DB5eHhQWVmZUPPatWtJLpc32k9MTAy9/fbbWv2NGjWK4uLiGh3rurE5deqUsPxh6xMRzZs3j9577z3h/syZM2natGla+w8PD6ePPvpIqG/SpEkNHnOdgoICkkqlwnF09epVkkqldPHiRWGdtLQ0kslkdOPGjQbbz5w5k6ZOnarV1qtXLyFsGqvv/uf3fjExMfTaa68J96urq0kmk1FKSopetd4/Xo9zvGk0GurVqxetXr1aqx4/Pz9KTk4monvH/9KlS4XlP//8M7m7u5NarX7gY6xTVVVFUyLep5OnU4iI6Oat2/T3qAV0924RERGlnDlHUyLeF04YNmzaSgv/EdNov8+Cp/ozm3PXCf85rgbQCsGe92YEHW2NDPYZjaWlJSIiIvDee+/hlVdewYABAzBs2DBYWlqirKwMN27cQGRkJKKjowHc+1ZUtVqN27dvo2PHjlp9OTg4ID4+Hr/88guKi4uhUqng5OSkVx2ZmZm4du0avL29hTalUok2bdoI983MzBpsZ2trK/xsYWEhTGm0bt0aZ86cwaJFi3D16lWoVCooFAqt6Z6HGT58OP71r38hKioKhw4dwuDBg9GqVStcvHgRRIQBAwYI66pUKtjb2zfoIyAgAH5+fggKCsKQIUMQHByMN954A8bGxjh16pTe/dTx8vJCXl4eAP3G2tT0jysZm/rcZGZmYuzYsVpt/v7+WlOb9ftvTGZmJiwtLeHl5SW0eXt7w9zcHFeuXIGzs7PW+teuXUNwcLBWm4nJHy99feprTP1jRyKRwNjYGAqFAgUFBU2qFXi8462kpAR37tzBSy+9pFVP9+7dtS6wqF+vpaUlVCoVamtrIZFIHtr/7fw7UKvVcHdzBQA4OXbAssWfALh3zJ04dRpdPN1hZHRvat/NVYbDR38BEQltz6qnOmwA3YEjRtCYmprC2NgYtbXa869KpRIAhIM0MjISEyZMwMGDB7FhwwYsX74cW7duhZ2dHQAgPj5e64UHAI6Ojg32FxUVBZVKhU2bNsHR0RHr1q3DunXr9K7Xz88PS5cu1WqzsrLSe/v60tPTMWPGDMTHx2PAgAEwNTVt0tVJgwYNQnR0NK5cuYKDBw9iyZIlwjIzMzPs3r1ba31dH8BLJBJs3LgR586dw5EjR7BgwQI4Oztjw4YNTeqnTv3nsalj3dT1zczMtH65A/cuBKg7dprKzMyswUUE9L+vbtfVp5GRkc6TC7Hqe5xagcc73uoe5/3PvVqtfiKPp+64MTNtOJ6ff/lvFBTcxYK5H/1Rj8QMarUaGiK0esbDpkVcIFAXOCm5JNo7GlNTU3Ts2BHp6ela7enp6bCwsICTkxOys7MRFxeHTp06ITw8HMnJyWjdujV27twJW1tbODg44MaNG3BxcRFu9vb2DV7oAHD8+HGMHz9eCKKHffh7/xmTq6srsrOz0b59e2E/Dg4ODz3Tf5hTp05BKpViyJAhMDU1hUql0nrhNnbGZmVlhX79+uGzzz5DdXU1AgICAABubm5QKpUoLy8X6nz++ed11nnkyBFs3boVvr6+mD17Nn744QekpKQgNTVVr37qX6AAAGlpaXBxcQHQtLHWZ/37x8Pd3R3nzp3Tajtz5gy6dOny0P08rL+KigpkZ2cLbenp6aipqdHZp1QqxeXLl7XaiP54fTRW3+OcketT6/39P87xZm1tjQ4dOuC///2v0FZTU4P09HS9x/th2v5vdiD/vr/pybt+A+kZmYiYGgYbG2uhPT//Dlrb2qIVX2HYMsIG+CNwxJw6mzp1KlauXIktW7bg0qVL2LVrF5YtW4bw8HAYGxvD2toaq1atwueff47r16/j+PHjyM/Ph0wmAwBERETgyy+/xK5du3D9+nWsWLECY8eO1Xrh15FKpfjuu++QmpqKbdu24euvv37gmVmbNm0gl8tx/PhxyOVyvPrqqzA1NcXf/vY3ZGVlITU1Fa+99hr27t37SI+7c+fOyMrKQnJyMs6ePYsPP/wQCoVCqOf+/esyfPhw7Nu3D4MGDRLCVSaTYeDAgZg9ezZSUlKQk5ODGTNmYPHixQ22VyqV+Pjjj7F7927cvHkTu3fvhomJCTp27KhXPydPnkRSUhLy8vIQFxeHq1evYsyYMU0ea33Wt7OzQ2ZmJtLS0qBSqfCXv/wF+/fvx5o1a/D7778jISEBx44dwzvvvKPX+Nva2sLY2BiHDx9Gfn4+XFxcEBISgjlz5uDChQs4f/48oqKiMHjwYHTq1KnB9pMmTcKePXtw6NAhVFVVYcOGDbh165awvLH69Hl+H0SfWu8fr8c93t5991189tlnOHToELKzsxEVFQWJRILhw4c3qXZd7O3bwbFDe/x24pRWu11rW7wXEQ6ptLPQRkQ4cfI0vLtqz2Q8q1pM2AD3AkfMz2gmTZqEqKgorF69GqNGjUJ8fDzCw8Mxa9YsAPf+uGzVqlU4fPgwBg0ahMjISERERGDYsGEAgHfeeQfh4eGIiYnBoEGDcPToUcTFxek8U/v0009RXl6ON954Azt27MCsWbNgaWmp85fgyy+/jN69eyMiIgJnz56FpaUlEhMTIZfLERoaimnTpmHIkCEYPHjwIz3ugQMHIiIiAjExMfjwww/RvXt3BAUFoby8XOf+dQkODoa5uTmGDh2q1b58+XJ07doV4eHhePXVV2FiYiKMZ31DhgzBRx99hKVLl2LgwIHYuHEjVqxYgfbt2+vVT/fu3XHixAmMGjUK27dvF96BNnWs9Vl/zJgxMDIywuTJk1FeXg4vLy+sWLECmzZtQkhICH744Qd88803Wp+pPYyZmRmmTZuGr776CmvXrgUALF26FG5ubpg8eTLCwsLg4+OD+Ph4ndu/9NJLiIyMRGRkJAIDA5Geng43NzdheWP16fP8Pkxjtd4/Xo97vL355psICwvDvHnzEBoaiuLiYmzatAmWlpZNrl2X4cNC8POxX3Hp8h+fAV2/cQsrE77Vmp7dd+Agrt+4iWEhj/a6a2mMSssrqP6Zdd3PddMOGo1GmGNVq1RQazRQqVTCraamBrW1tfD38/3TipZGFTz2/9psZmKE35c894QqYs1ZbGwssrKykJiYaOhSWAvxdcJqpJ6/gHfC3sDL/i9pLVOpVPhxz378sGs3Jk0Yh0EDgwxUZfPyVF4gsGSM7RP58jTGGHsUf5kahk1bkvB1wmrs3rMfXV/wgqWlJUpKSpB6IQ0VFRV4+81JeKVfoKFLbTaeyrAZ72/O37LJGDOYVsbGeGPi6+gXGIBffv0NWVeyUVVdDRsba/TtE4BX+gWiTRs7Q5fZrDyV02iMMcaeLi3qAgHGGGPNE4cNY4wx0XHYMMYYEx2HDWOMMdFx2DDGGBMdhw1jjDHRcdgwxhgTHYcNY4wx0XHYMMYYEx2HDWOMMdFx2DDGGBMdhw1jjDHRcdgwxhgTHYcNY4wx0T2V32djKDk5OdizZw/y8/NhY2ODfv36ISAgwNBlAQBOnToFmUwGBwcHQ5fCGGMNcNjoqbKyEomJiRg7dixeeOEFFBQUYO3atbCxsUG3bt1E3TcRwcjI6KHrnDp1Cra2tk0Km7rvLmqsb8YYe1wcNnoqKiqCsbExfHx8AADOzs4ICQmBQqEAcO9L5nbv3o2UlBRYW1tj5MiR8PLyQn5+PlavXg0XFxfk5ubCxsYGEyZMEEIhJycH27dvh0KhgFQqxYQJE2Bubo78/HwkJCSgU6dOuH37NqKjox+47pIlSyCXy7F582YMGTIEAQEByM3NxY4dO1BUVAQXFxeMHTsWdnb3vjlw0aJF6NKlCzIyMjBr1iy0a9fOMIPKGHtm8Gc2enJ0dIS1tTWSkpJQXFwMAPD19UWvXr0AAL/88gtu3ryJ6OhojB8/Hlu2bEFVVRUAoLy8HH369MHcuXPh5eWFbdu2AQAqKiqQmJiIyZMnY8GCBVAqlTh27JiwT4VCgW7dumH27NkPXTcqKgpOTk6YMGECAgICUFVVhTVr1iAoKAgff/wxXFxcsG7dOtT/RlYLCwtER0ejbdu2f8r4McaebRw2ejIxMcH7778Pc3NzfPXVV1ixYgUuXbokLD979iyCgoJgYWEBFxcXODs74+rVqwDu/WLv3LkzAKB///7Izc1FdXU1JBIJPvjgAzg7O8PExASenp4oLCwU+rSysoKfnx/MzMwaXbe+S5cuwcHBAT4+PjAxMcHAgQNRUlKCgoICYZ3evXvD0tKSp9AYY38KnkZrAktLS4wYMQLDhg1DRkYGtm7ditGjR8Pb2xtlZWVYv3698Mtbo9Gge/fuDaaoTE1NYW5ujoqKCtjb2+PWrVtISkqCUqkUpsd0MTU11XvdiooKtGnTRrhvbGyM1q1bQy6Xo3379k9oNBhjTH8cNnq6cOECSkpK0L9/fxgbG6Nbt264efMmMjIy4O3tDRsbG4wbNw4uLi5a2+Xn52vdr66uRnV1NaytrXHz5k3s27cP06dPR+vWrXHs2DHk5eXp3H9T1rWzs0NaWppwX6PRoLS0FNbW1o85Cowx9mh4Gk1Pbdq0waFDh3Dx4kXU1taiuLgYly9fhqOjIwDAx8cHBw4cQGVlJSorK7Fr1y6UlpYCuHclW0ZGBtRqNX766Se4uLgI726ICESEwsJCnD9/HhqNRuf+G1vXzMwMRUVFUKvV8PDwwN27d5Geng61Wo2ff/4Z1tbWcHJyEn+gGGNMB35no6eOHTti4sSJ+Omnn7B+/XpYWVnhxRdfRGBgIAAgKCgIe/bswfLly6HRaODv7w9bW1tUVVVBIpEgLS0NW7duRbt27TBx4kQAgIeHBzw8PBAXF4c2bdrA09MTd+/e1bn/xtbt2bMnduzYAbVajf79+2PKlClITk7Gli1b4OzsjLCwMP58hjFmMEal5RVU/yqlup/rzpo1Gg2ICBqNBmqVCmqNBiqVSrjV1NSgtrYW/n6+BnkAzV3dJcwLFy40dCmMMWYwPI3GGGNMdBw2jDHGRMfTaIwxxkTH72wYY4yJjsOGMcaY6DhsGGOMiY7/zqYJamqUKCqWQ1FZZehSGGPNmJWlBdq1tYNEYmboUpoNDhs91dQokXfjNhzs28LJ8TlDl8PYM+9KTi7cXV0aX9EA5KXlyLtxG52ed+TA+R+eRtNTUbEcDvZtYdfaxtClMMaaObvWNnCwb4uiYrmhS2k2OGz0pKis4qBhjOnNrrUNT7nXw2HDGGNMdBw2jDHGRMdhwxhjTHQcNowxxkTHYcMYY0x0HDaMMcZEx2HDGGNMdBw2jDHGRMdhwxhjTHQcNowxxkTHYcMYY0x0/L8+M8aeCUlJSaitrRXujxs3DiYmJqiursaOHTuEdolEgtGjRwMAUlJSkJ2dLSzz9/eHm5vbn1d0C8Jhwxh7JtTU1ECpVDZoJyJUV1fr3EalUmktU6vVotXX0nHYMMZarLKyMlRWVgK4Fyr13blzB61atUJNTY1Wu0ajQX5+PgAI29YpLS0Vlj333HMwNuZPIvTFYcMYa7HS0tKQlZWlc9mBAwd0tiuVSuzdu1fnsoyMDGRkZAAAJk2aBIlE8mQKfQZwLDPGGAAbGxuYmPD5t1g4bBhjDED//v1hb29v6DJaLA4bxhhjouP3jIyxZ1arVq0a3K9r02g0DS4qYI+Ow0ZkiYmJ6N27Nzw9PVFZWYlVq1YhLCwMtra2hi6NsWdeaGio8CG/RCLBK6+8IlzefPr0aVy9etWQ5bUoHDYi27hxI8zNzeHp6YmCggKsX78egwcP5rBhrBmo/8ecI0aMwJkzZ4RLm9mTxWEjsvqXV3bu3BmnT582YDWMMWYYfIEAY4wx0fE7G8YYA7B3715oNBpDl9FicdgwxlosT09PODk5idK3qampKP22VBw2jLEWy97env9Qs5ngz2wYY4yJjsOGMcaY6DhsGGOMiY7DhjHGmOg4bBhjjImOw4YxxpjoOGwYY4yJjsNGT1aWFpCXlhu6DMbYU0JeWg4rSwtDl9FscNjoqV1bOxTeLebAYYw1Sl5ajsK7xWjX1s7QpTQb/D8I6EkiMUOn5x1RVCxH4d1iQ5fDGANwJSfX0CXoZGVpgU7PO0IiMTN0Kc0Gh00TSCRmcHJ8ztBlMMbYU4en0RhjjImOw4YxxpjoOGwYY4yJjsOGMcaY6DhsGGOMiY7DhjHGmOg4bBhjjImOw4YxxpjoOGwYY4yJjsOGMcaY6DhsGGOMiY7DhjHGmOg4bBhjjImOw4YxxpjoOGwYY4yJ7v8B5QR/rDHF1T0AAAAASUVORK5CYII=)

Now, you can denote nesting by using 2 dots, and the example path above can be denoted as `user_data.address..city`.

## Snowplow Event Mapping Options

This section includes the mapping rules that concern a Snowplow event as claimed by the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/):

### Snowplow Atomic Properties Rules

This option indicates if all Snowplow [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) properties of the event should be included in the JSON body. By default this option is disabled.

If enabled, an additional text field optionally allows you to specify a key under which those atomic properties will be nested. Leaving it blank adds those properties in the request body without nesting. Dot notation can also be used here.

As an example, this section configured as:

![The Snowplow Atomic Properties Rules section with \&quot;Include all Snowplow Atomic Event Properties\&quot; enabled and the nesting key set to \&quot;snowplow\_atomic\&quot;](/assets/images/snowplow_atomic_nest-4975fe7246e9a491c69b3042e45b280c.png)

will result in the following JSON structure:

```json
{

    ...,

    "snowplow_atomic": {

        "app_id": "fooBar",

        "platform": "mobile",

        ...

    },

    ...

}
```

Please, note that some of the Snowplow atomic properties are already mapped to [common event properties](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) by the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/).

### Snowplow Self-Describing Event Rules

This option indicates if the Snowplow Self-Describing data will be included in the request body and it is enabled by default.

Similarly to the above section, you can also specify a key under which the self-describing data will be nested. Leaving it blank adds those properties in the request body without nesting. Dot notation can also be used here.

As an example, this section configured as:

![The Snowplow Self-Describing Event Rules section with \&quot;Include Self Describing event\&quot; enabled and the optional nesting key left blank](/assets/images/snowplow_self_desc_no_nest-c7502209e3ea302074ec4427c13b5b6d.png)

will result in the following JSON structure:

```json
{

    ...,

    "self_describing_event_com_acme_test_foo_1": {

        "mySelfDescProp": "exampleValue",

        ...

    },

    ...

}
```

### Snowplow Event Context Rules

This section describes how the HTTP Request tag will use the context Entities attached to a Snowplow Event.

![The Snowplow Event Context Rules section showing \&quot;Extract entity from Array if single element\&quot; checked, entities set to include \&quot;All\&quot;, and empty tables for entity mapping and exclusion rules](/assets/images/context_rules-9da59ffc2b9d28034d92e7d5a9013550.png)

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in request body

Using this drop-down menu you can specify whether you want to Include `All` (default) or `None` of the Snowplow context entities in HTTP Request's body.

#### Nest all unmapped Entities under key

This option is available only if the previous option ([Include Snowplow Entities in request body](#include-snowplow-entities-in-request-body)) is set to `All`.

It applies **only** to unmapped entities, i.e. all included entities whose mapping is not edited in the following ([Snowplow Entities to Add/Edit mapping](#snowplow-entities-to-addedit-mapping)) table.

With this setting you can specify a key under which the Snowplow event's unmapped entities will be nested. Alternatively, leaving it blank adds the unmapped entities in the request body without nesting. You can also use dot notation in this value.

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- **Entity Name**: The Entity name to add/edit-mapping (required).¹
- **Destination Mapped Name**: The key you could like to map it to in the request body (optional: leaving the mapped key blank keeps the same name). You can use dot notation here as well to signify further nesting. This value is independent of the [nesting of unmapped entities](#nest-all-unmapped-entities-under-key) setting above.
- **Apply to all versions**: Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if [Include Snowplow Entities in request body](#include-snowplow-entities-in-request-body) is set to `All`), you can specify the context entities you want to exclude from the HTTP Request body. In its columns you can specify:

- **Entity Name**: The Entity name (required).¹
- **Apply to all versions**: Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

**pre-v0.2.0**

#### Snowplow Event Context Rules

This section describes how the HTTP Request tag will use the context Entities attached to a Snowplow Event.

##### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

##### Include all Entities in request body

Leaving this option enabled (default) ensures that all Entities on an event will be included within the request data.

Optionally, you can also specify a key under which the Snowplow event's contexts will be nested. Alternatively, leaving it blank adds all entities in the request body without nesting.

Disabling this option, reveals the options so that individual entities can be selected for inclusion. Using the "Snowplow Entity Mapping" table, these entities can also be remapped to have different names in the JSON body of the request. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1` where `com_snowplowanalytics_snowplow` is the event vendor, `web_page` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/webPage/jsonschema/1-0-0`

##### Include unmapped entities in request body

This option enables you to ensure that all unmapped entities (i.e. any entites not found in the "Snowplow Entity Mapping" rules above) will be included in the request body.

Again, optionally, you can also specify a key under which the Snowplow event's unmapped entities will be nested. Alternatively, leaving it blank adds the unmapped entities in the request body without nesting.

## Additional Event Mapping Options

If you wish to pick other properties from the Client event and map them into the request body, this can be specified in this section.

### Event Property Rules

#### Include common event properties

Enabled by default, this option sets whether to include the event properties from the [Common Event definition](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) in the request body. Inclusion of the `user_data` property is not affected by this setting (see next option).

Also, you can optionally specify a key under which the Common Event properties will be nested. Alternatively, leaving it blank adds the the common event properties in the request body without nesting.

#### Include common user properties

Disabled by default, this option sets whether to include the `user_data` properties from the common event definition in the request body.

Again, you can optionally specify a key under which the `user_data` properties from the common event will be nested. Alternatively, leaving it blank adds the `user_data` in the request body without nesting.

#### Additional Event Property Mapping Rules

Using this table, you can additionally specify the Property Key from the Client Event, and then the key you could like to map it to (or leave the mapped key blank to keep the same name). You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (note the array index 0) or pick non-Snowplow properties if using an alternative Client.

## Additional Request Data

This section allows you to add custom properties in the request body that are "external" to the event, in other words it provides the ability to add custom constant or variable request data.

## Post-processing

![The Post-processing section with subsections for JSON Stringify and Encode base64url, each with an empty table for specifying property names or nested paths to transform](/assets/images/post_processing-706f515d594c968ce8cfbd595f0bbd6e.png)

This section provides a way to easily configure some basic post-processing of values in the constructed HTTP request payload. The order of the subsections denotes the post-processing order. For more advanced use cases you can still use the **Additional Request Data** section above and provide values through GTM server-side variables.

### JSON Stringify

In this table you can specify the property names or paths of the HTTP request payload whose values you want to transform into JSON strings. Dot notation can also be used to denote nested paths.

### Encode base64url

In this table you can specify the property names or paths of the HTTP request payload whose values you want to encode to base64url. Encoding is only applied to string values. Dot notation can also be used to denote nested paths.

## Request Headers

Similarly to the above, this section allows you to add custom headers to the HTTP request towards your custom endpoint.

## Additional Options

Finally, this section offers two additional configuration options:

- Changing the HTTP request method from POST (default) to PUT.
- Changing the default request timeout (5000 seconds)

## Logs Settings

Through the Logs Settings you can control the logging behavior of the HTTP Request Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the HTTP Request GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{

    "Name": "HTTP Request", // the name of the tag

    "Type": "Message",      // the type of log (one of "Message", "Request", "Response")

    "TraceId": "xxx",       // the "trace-id" header if exists

    "EventName": "xxx"      // the name of the event the tag fired at

}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# HTTP Request Tag for GTM Server Side
> Send Snowplow events to custom HTTP endpoints from GTM Server Side using the HTTP Request Tag with flexible request configuration and authentication options.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/http-request-tag-for-gtm-ss/

The HTTP Request Tag for GTM SS allows events to be forwarded to any JSON HTTP endpoint. This Tag works best with events from the Snowplow Client, but can also work with other GTM SS Clients such as GAv4.

## Template installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click “Search Gallery” in the Tag Templates section
2. Search for “HTTP Request” and select the official “By Snowplow” tag
3. Click "Add to Workspace"
4. Accept the permissions dialog by clicking “Add”

### Manual Installation

1. Download `template.tpl` – Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select "Save Link As…"
2. Create a new Tag in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

## HTTP Request Tag Setup

With the template installed, you can now add the HTTP Request Tag to your GTM SS Container.

1. From the Tag tab, select “New”, then select the HTTP Request Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to your custom destination.
3. Enter your destination URL
4. [Configure the tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/http-request-tag-for-gtm-ss/http-request-tag-configuration/) to construct the desired JSON request body
5. Click Save

---

# Forward events with Google Tag Manager Server Side
> Send Snowplow events to multiple destinations using Google Tag Manager Server Side with Snowplow-authored tags and vendor/community tags for flexible event routing.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/

To support sending events to adjacent destinations, Snowplow works with Google Tag Manager Server Side. There are both Snowplow Authored and Vendor/Community authored Tags which will allow event data to be forwarded to different destinations.

Before reading this documentation, we recommend you become familiar with [the fundamentals of Server Side tagging](https://developers.google.com/tag-platform/tag-manager/server-side/intro).

Taking an approach with Snowplow and GTM SS offers many additional flexibility and control:

- Full visibility into all transformations on the data
- Ability to evolve sophistication over time
- All data remains in your private cloud until you choose to forward it
- Ease of setup due to rich libraries of tags and familiar GTM UI

## Configuration Options

GTM SS with Snowplow can be setup in two different configurations.

![Diagram showing two GTM Server Side configurations: the Destinations Hub option where events flow from the Snowplow pipeline through Snowbridge into GTM Server Side before being forwarded to destinations like Braze and Amplitude; and the Server Side Tag Manager option where a tracker sends directly to GTM Server Side which routes to Snowplow, Google Analytics, and other destinations](/assets/images/gtmssoptions2-b962d1cc761a54c53f1c793e66d8ceec.png)

### Destinations Hub (Post-pipeline)

Use GTM SS to relay enriched events to destinations. Events are sent to GTM SS via Snowbridge after being processed by your Snowplow pipeline.

- For Snowplow CDI, you can [request setup](https://console.snowplowanalytics.com/destinations) through Console
- For Snowplow Self-Hosted, see [Snowbridge](/docs/api-reference/snowbridge/)

> **Note:** Destinations Hub is the recommended way to set up GTM Server-Side because it allows you to take full advantage of the Snowplow pipeline, and forward validated and enriched data to downstream destinations.

### Server Side Tag Manager (Pre-pipeline)

Use GTM SS to relay raw events before the Snowplow pipeline to destinations, including to your Snowplow pipeline. This is useful if your company uses GA but does not want to send data to Snowflake or Databricks via BigQuery.

### Principles for AWS deployment

> **Info:** GTM SS **should** be deployed into a different account to the Snowplow sub-account to maintain full segmentation of the infrastructure that Snowplow manages from that which is managed by the Snowplow customer.
>
> It would be possible to set up a separate VPC within the Snowplow sub-account but it is discouraged. VPC peering would be required to keep the traffic private otherwise traffic would go over public internet.
>
> GTM SS as a Destination Hub is normally intended to be public facing, however as a Server Side Tag Manager it could use both public internet routes and VPC peering to fan out traffic via private routes.

## Deploying Google Tag Manager Server Side

GTM SS must first be deployed before it can be used. This can easily achieved by deploying to GCP App Engine from directly within the GTM User Interface. Alternatively, docker images are available for other deployment options, such as on AWS or with Kubernetes.

- [App Engine Setup Guide](https://developers.google.com/tag-platform/tag-manager/server-side/script-user-guide)
- [Manual Setup](https://developers.google.com/tag-platform/tag-manager/server-side/manual-setup-guide)

## Snowplow Client

To receive events in your GTM SS container, the Snowplow Client must be installed. This works for both events direct from the tracker, or enriched events from the pipeline.

The Snowplow Client populates the common event data so many GTM SS tags will just work, however it also populates a set of additional properties to ensure the rich Snowplow event data is available to Tags which wish to take advantage of this, such as the Snowplow Authored Tags.

## Snowplow Tag

If using GTM SS as a Server Side Tag Manager for Snowplow JavaScript Tracker events, you will want to ensure you forward these events to your Snowplow Collector. The Snowplow Tag will automatically forward any events the Snowplow Client receives once it has been configured with your Collector URL. It can also construct Snowplow events from other GTM SS Clients such as GAv4.

## Snowplow Authored Tags

Snowplow have created a number of GTM SS Tags which work best with the Snowplow Client and make use of the rich data available from Snowplow tracker or Enriched events. See the tags for:

- [Amplitude](/docs/destinations/forwarding-events/google-tag-manager-server-side/amplitude-tag-for-gtm-ss/)
- [Braze](/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/)
- [Iterable](/docs/destinations/forwarding-events/google-tag-manager-server-side/iterable-tag-for-gtm-ss/)
- [LaunchDarkly](/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/)

---

# Iterable Tag for GTM Server Side
> Forward Snowplow events to Iterable from GTM Server Side using the Iterable Tag for cross-channel marketing automation and customer engagement.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/iterable-tag-for-gtm-ss/

The Iterable Tag for GTM SS allows events to be forwarded to Iterable. This Tag works best with events from the Snowplow Client, but can also construct Iterable events from other GTM SS Clients such as GAv4.

The tag is designed to work best with Self Describing Events, and atomic events, from a Snowplow Tracker, allowing for events to automatically be converted into an Iterable events, include Iterable Identity events. Additionally, any other client event properties can be included within the event properties or user properties of the Iterable event.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

There are two methods to install the Iterable Tag.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click “Search Gallery” in the Tag Templates section
2. Search for “Iterable” and select the official “By Snowplow” tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking “Add”

### Manual Installation

1. Download [template.tpl](https://raw.githubusercontent.com/snowplow/snowplow-gtm-server-side-iterable-tag/main/template.tpl) - Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select “Save Link As…”
2. Create a new Tag in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

## Iterable Tag Setup

With the template installed, you can now add the Iterable Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the Iterable Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to Iterable
3. Enter your Iterable API Key for a Standard Server Side integration. This can be generated from Iterable "Integrations -> API Keys" settings page
4. Click Save

---

# Configure Iterable Tag for GTM Server Side
> Configure user identifiers, identity events, entity mapping, and event properties for the Iterable Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/iterable-tag-for-gtm-ss/iterable-tag-configuration/

## Iterable API Key (Required)

Set this to the API of your Iterable HTTP API Data Source.

Iterable provides four different types of API keys, each of which can access a different subset of Iterable's API endpoints. For the endpoints currently in use (`events/track` and `users/update`) the Javascript type key has enough permissions. The Mobile and Standard key types have more permissions than the Javascript type, so can also be used.

## Identity Settings

Iterable requires users to be identified to work best. The options in this section configure how you wish to identify users to Iterable based on your Snowplow events.

### Identifiers

#### Use client\_id for anonymous users

Specify whether `client_id` is used to create a placeholder email for anonymous users. This is useful for implementations where there is no identifiers for a user besides device identifiers (such as Browser Cookies).

#### email

##### Use email\_address from common user data

For Snowplow Tracking, the common user data can be populated by using the `iglu:com.google.tag-manager.server-side/user_data/jsonschema/1-0-0` context entity. This schema is available on [Iglu Central](https://github.com/snowplow/iglu-central/blob/853357452300b172ebc113d1d75d1997f595142a/schemas/com.google.tag-manager.server-side/user_data/jsonschema/1-0-0).

This option is enabled by default. Disabling it allows for any other properties of the event to be selected for the `email` property on the Iterable event.

##### Specify email

As mentioned above, this table is revealed when disabling the "Use email\_address from common user data" configuration option. Using this table allows you to specify key paths to look for the `email` value. You can also set the search priority to denote the preference for the value to use.

The columns of this table are:

- **Search Priority**: The priority of a key path when looking for `email` (higher number means higher priority).
- **Property Name or Path**: The key path to look for in the server-side common event.

#### userId

##### Use user\_id from common user data

Iterable can also accept a User Id, rather than the preferred e-mail address. Enabling this property will use the `user_id` property from the server-side common event as the `userId` identifier of the user.

##### Specify userId

This table is revealed when disabling the "Use user\_id from common user data" configuration option. Using this table allows you to specify key paths to look for the `userId` value. You can also set the search priority to denote the preference for the value to use.

The columns of this table are:

- **Search Priority**: The priority of a key path when looking for `userId` (higher number means higher priority).
- **Property Name or Path**: The key path to look for in the server-side common event.

As an example of how Search Priority works, according to the following setup, in order to set the value for Iterable's `userId`, the Tag will first look for `user_id` in common event. If that is not found, then it will use the value of `user_data.email_address`:

![The userId Specify userId table showing search priority 20 mapped to \&quot;user\_id\&quot; and search priority 10 mapped to \&quot;user\_data.email\_address\&quot; as a fallback](/assets/images/user_id_example-c2aa0f9095a8b50674d151370535eb2b.png)

### Identity Events

#### Use the default `identify` event

Iterable allows for user information to be updated once a user has identified themselves (for example, to update their placeholder email to their real email address).

To Identify a user to Iterable, you can send a Self Describing `identify` event. This schema is [available on Iglu Central](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/identify/jsonschema/1-0-0).

For example, using the JavaScript Tracker v3, this would look like:

```javascript
window.snowplow('trackSelfDescribingEvent', {

  schema: 'iglu:com.snowplowanalytics.snowplow/identify/jsonschema/1-0-0',

  data: {

    id: '2c5ba856-ee07-47b5-a3a6-63100026ed63',

    email: 'john.doe@example.com'

  }

})
```

If you would like to specify your own event, disabling this option allows you to select your own event name and properties which can used to fire identity updates to Iterable.

#### Specify identity event(s) by event name

This multi-line text box is revealed when disabling the "Use the default `identify` event" configuration option above.

In general, "identity events" are the event names which will make the Iterable Tag call the `/users/update` [API endpoint](https://api.iterable.com/api/docs#users_updateUser) (create or update a user), using the identifiers and the user\_data specified by the tag configuration. These events might be different than the default Snowplow Identify schema, for example sign\_up, login etc, from your own custom event schemas.

## Snowplow Event Mapping Options

### Include Self Describing event

Indicates if a Snowplow Self Describing event should be in the `dataFields` object.

### Snowplow Event Context Rules

This section describes how the Iterable tag will use the context Entities attached to a Snowplow Event.

![The Snowplow Event Context Rules section with \&quot;Extract entity from Array if single element\&quot; checked, entities set to include \&quot;All\&quot;, and empty tables for entity mapping and exclusion](/assets/images/context_rules-3b86d3660566214e58f2854ec459ac3c.png)

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in event properties

Using this drop-down menu you can specify whether you want to Include `All` or `None` of the Snowplow context entities in Iterable's within the Event Data fields of the Iterable event.

If disabling this, individual entities can be selected for inclusion. These entities can also be remapped to have different names in the Iterable event, and can be included in either event data or user data. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_webPage_1` where `com_snowplowanalytics_snowplow` is the event vendor, `webPage` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/webPage/jsonschema/1-0-0`

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- The Entity name to add/edit-mapping (required).¹
- The key you could like to map it to (optional: leaving the mapped key blank keeps the same name).
- Whether to add in event data fields or user data fields of the Iterable event (default value is `event data`).
- Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if `Include Snowplow Entities in event properties` is set to `All`), you can specify the context entities you want to exclude from the Iterable event. In its columns you can specify:

- The Entity name (required).¹
- Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

**pre-v0.2.0**

#### Snowplow Event Context Rules

##### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

##### Include all Entities in event\_properties

Leaving this option enabled ensures that all Entities on an event will be included within the Event Data of the Iterable event.

If disabling this, individual entities can be selected for inclusion. These entities can also be remapped to have different names in the Iterable event, and can be included in either event data or user data. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_webPage_1` where `com_snowplowanalytics_snowplow` is the event vendor, `webPage` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/webPage/jsonschema/1-0-0`

##### Include unmapped entities in event\_properties

If remapping or moving some entities to User Data with the above customization, you may wish to ensure all unmapped entities are still included in the event. Enabling this option will ensure that all entities are mapped into the Iterable event.

### Additional Event Mapping Options

If you wish to map other properties from a Client event into an Iterable event they can be specified in this section.

#### Event Property Rules

##### Include common event properties

Enabling this ensures properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Iterable Event Data.

##### Additional Event Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client.

#### User Property Rules

##### Include common user properties

Enabling this ensures user\_data properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Iterable Event Properties.

##### Additional User Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client.

## Advanced Event Settings

### Merge user dataFields

Enabling this option will merge the user dataFields when updating an Iterable user, instead of replacing with the new user data, which is the default behavior.

## Logs Settings

_(Available since v0.2.0)_

Through the Logs Settings you can control the logging behavior of the Iterable Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Iterable GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{

    "Name": "Iterable",  // the name of the tag

    "Type": "Message",   // the type of log (one of "Message", "Request", "Response")

    "TraceId": "xxx",    // the "trace-id" header if exists

    "EventName": "xxx"   // the name of the event the tag fired at

}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# LaunchDarkly Tag for GTM Server Side
> Forward Snowplow events to LaunchDarkly from GTM Server Side using the LaunchDarkly Tag with metric import REST API for experiment tracking.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/

The [LaunchDarkly Tag for GTM SS](https://github.com/snowplow/snowplow-gtm-server-side-launchdarkly-tag) allows events to be forwarded to LaunchDarkly using its [metric import REST API](https://launchdarkly.com/docs/api/metrics#importing-metric-events). This Tag works best with events from the Snowplow Client, but can also work with events from other GTM SS Clients such as GAv4.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

Coming soon!

## LaunchDarkly Tag Setup

With the template installed, you can now add the LaunchDarkly Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the LaunchDarkly Tag as your Tag Configuration.
2. Select your desired Trigger for the events you wish to use as metrics in LaunchDarkly experiments.
3. [Configure](/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/launchdarkly-tag-configuration/) the Tag.
4. Click Save.

---

# Configure LaunchDarkly Tag for GTM Server Side
> Configure metric type, authentication, context keys, and event creation time for the LaunchDarkly Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/launchdarkly-tag-configuration/

This is the configuration of the LaunchDarkly Tag.

## Event Name (Required)

This is the value that identifies your metric. When you create your metric in LaunchDarkly, its Event name must exactly match this value.

## Metric Type

![Metric Type configuration section showing an Event Name text field and two radio button options: Conversion and Numeric Metric, with Numeric Metric selected by default.](/assets/images/01-metric-type-35a667b323c01a8c56eb17ec207d2432.png)

This section allows you to select the Metric Type for this Tag. The options are:

1. **Numeric Metric** (default): Choosing this metric type reveals the [Metric Options](#metric-options) configuration section, where you must specify a Metric Value.
2. **Conversion**: Conversion Metrics should just be triggered and the metric name will be sent as a conversion.

## Authentication

![Authentication section with two empty text input fields: Project Key and Environment Key.](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxYAAADPCAYAAABoZ0fdAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d17VFTnvf/xDwwwckcEBUFAGLyAl3ghiVhv1SSmamKtVmNizMXk9PR3fr+2Oa3V2pWY1XpO0npWe3659ZdaY6JRiVZP4l2DJire0GgkGAREoxgVhXCR2wx75veHdSKVROKGDJD3ay3Xmnn2s5/9ncE/+PA8z95eJVdKXYZhyDAMOQ1DhtOQw9Egh90uwzBUV1+vtCGDBQAAAABfxdvTBQAAAABo/wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWt7B582bNnTtXR48eNT3WxYsXNXfuXGVmZrZAZc2zd+9eLVy4UNXV1e1yfAAAALQPBItbOH78uCTp2LFj3/jc119/Xc8++2xLl/S15s2bpzfffNP9PiAgQJ07d5bFYjE99pkzZzR37lzt2bOnVcYHAABA++Xj6QJu5cyZMyooKNA999zT5PEdO3YoOTlZCQkJLX7ts2fPqqysTBEREcrPz1dtba38/f1b/DqtafDgwRo8eHC7HR8AAADtg+VXc3+90OVy6cZ/TqdTTsOQy+VSg2Eopnu0xwr8+9//rkOHDskwDNlstkbHtmzZoszMTFVVVWnQoEEtfu3du3fr/PnzmjFjho4cOaLIyEjFxMS4jz/77LM6deqU+xfrXbt26bXXXlNqaqr++Mc/qqSkRA0NDe7w4+Pjo/379ysyMlJZWVlav369cnNzlZiYqMDAQElSXV2d1q1bp4yMDGVlZamsrExJSUmyWCzat2+fXnrpJfn7+ysjI0NbtmzRuXPn1LdvX5WUlOh3v/udXC6XLl++rMzMTI0bN04bN27UkiVLNHToUPn7+6uhoUEbNmxQRkaGdu7cqeLiYvXs2VNWq1WSdPToUa1cuVIbN27URx99pKCgIEVFRWnPnj16++23JUn5+fk6ffq0hgwZctP4LpdL77//vlavXq2tW7fq1KlTio2NVVBQkCRp4cKFOnPmjPLz87Vq1SodOHBAnTt3VteuXVv85wcAAIBvT5tfCjVr1iwlJCRo586d2rp1q7t9y5Yt2rVrlxISEvTII4+0yrWPHz8um82m3r17KyQk5Bsth3rooYcUHR0tX19fzZ49u9EvzocOHVJkZKTS0tJUXFysjRs3uo+99dZbysnJ0YgRI3THHXfowIEDjT63JO3cuVMDBgyQzWZTbm6uDhw4oPDwcM2ePVteXl7q0aOHHn300Sbr2rp1q7KystSvXz+NGDFC+fn5WrFihSSpqKhIq1atUlhYmKZMmaLAwECtWrVKJSUlSklJ0f333y9JSktL03333dfk+Nu2bdOOHTvUs2dPjR07VhcuXNCSJUtkt9vdfU6cOKGamhqNGjVKdrtda9askWEYzf5uAQAA0Pa0+aVQfn5+mjNnjpYsWaKdO3dKkpxOpz744AMlJCRozpw58vPza/Hrnj59WhUVFRo3bpwkKSUlRYcOHVJ1dbV7duHr9O3bV3v27NEXX3yh1NRUSVJlZaUkacSIEZowYYKka8utiouLJUnFxcUqLCzUhAkTNGzYMElSWVmZjhw5okmTJrnHnjp1qlJSUuRwOPTb3/5WxcXFGjVqlFJTU+Xl5aWQkBClpKTcVJNhGNq3b59SU1P1ox/9SJLk6+urPXv2qKamRqGhoZo9e7YSExPl7++v8PBw/eUvf9GZM2d05513qmfPnpKkqKgoxcfH3zR+Q0ODsrKylJSUpJkzZ0qSYmJitHTpUh0+fFjp6enutscee8x9zq5du1RaWsqsBQAAQDvW5oOF1HS4aM1QIX25Wbtbt24qLS1VbGysDhw4oOPHj7t/6b9dnTp1avS6vr5ekvT5559LkjZt2qRNmzY1Oud6H+nahmnpWiiwWCyNZgO+TllZmRoaGhot5xo9erRGjx7tHreoqEhLlixRWVmZe9wbr32r8evr690BRJKSkpIkSefPn3e33bhP5fp30dzPAAAAgLapXQQLqXG4kNSqocLpdConJ0eS9OqrrzY69vHHH5sOFrcyfvx4JSYmNmrz9fU1Pa6399evfMvJydGaNWuUnp6uqVOn6sqVK1q+fHmzx/fxufbfycvL66ZjTqfzmxULAACAdqXdBAvpy3Bx/XVrOXXqlK5evaphw4Y12jCenZ2tkydPqrKyUiEhIbJaraqrq3Mf/+e/7Ht5ecnlcjX7ulFRUZKk8vJy912uampqVFtbe8tQcKOvumZYWJh8fHzcMyPStZmZo0ePavr06SosLJQk3XPPPQoMDFRVVdU3Ht9qter06dPutuuvo6M9dwMAAAAAtL52FSyk1g0U111fBjV69Gh17tzZ3e7l5aW8vDx9/PHHGjFihGJjY3XixAlt3LhRDQ0NOnjwYKNxgoODVV9fr82bN+uuu+665XXj4uJks9l08OBBOZ1ORUZG6tChQwoNDdXTTz/drNpDQkJ05swZZWZmasyYMY2OWSwWDR8+XLt379a6desUGhqq3bt3KzIyUgEBAYqMjJQkrVu3TrGxsdq7d6+kL2cbQkJC3N9PQECAhg4d2mh8b29vjRw5Ujt27NDKlSsVFRWlrKwsBQQEaMiQIc2qHwAAAO1Tm78r1LfNMAx98sknioqKahQqJKlXr17y8fHRxx9/LEmaOHGi4uLidODAARUXF98UHkaNGqWIiAjt37/fvXH7Vh555BENGjRIOTk52r59u7p06eLeaN0c48ePl3TtidhNzSzcd999Sk9P1/Hjx7Vr1y7ZbDb3HaSGDRumu+++W4WFhTpw4IBGjBgh6dreCUkKDw9Xenq6SkpK9MknnzR5/bFjx2rMmDEqLCxUZmamunbtqp/85CfN2vAOAACA9sur5EqpyzAMGYYhp2HIcBpyOBrksNtlGIbq6uuVNoQHoAEAAAD4asxYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEzz8XQBaH8qKyt1Mr9QFRUV6uTvr54JcYrp3t3TZQEAAMCDCBZotsqqKq3KWKsDB7NlsVgUEhKsmppa1dXVqWdCvGY9PEOJPRM8XSYAAAA8gGCBZim5fFkv/vHPslgs+ulP5mjQwAHy8fGRy+VS0ekzWv/uBv3Hi/+lf336CQ0ZPMjT5QIAAOBbZvnV3F8vdLlcuvGf0+mU0zDkcrnUYBiK6R7t6TpbldPp1K9//Wvt2LFDO3bs0O7du3Xq1CnFxsYqKCjoG4+3ePFiJScnKzAw8LbqMQxDmZmZSkhIkLf3zdtgXn31VVksFkVHX/u5nDx5Uq+88opSU1Nv+5pfp6GhQf/5xz8pMCBAv5n37+oaGaGly5Zr6ZvLlX34Iw0c0F8T7r9PVVevat3/vKdBdwxUSEhwi9cBAACAtovN2zeYP3++XnjhBS1YsEBxcXFatmzZTX1cLtctx/nxj3+s8PDw267DMAzt2LFDhmHcsu/Fixe1atUqPfzww4qMjLzta36dXR/uUWlpqf7P//qJAgMCtGJlhvILTunHP/qhQkND9OeXXlVdXb0enjFNPWJj9c7a9a1SBwAAANoulkLdwNvbW97e3urUqZPGjBmjzMxM1dTU6NixYzp27Jjq6+sVHR2tGTNm6LPPPtP69etVWlqq+Ph4TZ06VWFhYZKk5cuX6/HHH1f37t3ldDq1adMmZWdnKygoSJMmTVLfvn0lSfn5+XrvvfdUUVGhpKQkTZs2TZWVlfrrX/8qSXrhhRf05JNPKiYmpsl6q6qqtHTpUt1///2y2Wzu9sOHD2vbtm1yOp1KT0/X2LFjdfbsWS1ZskTPPfecLBaLCgoKtHbtWs2fP/+W38uBg9lKH3a3One+9vnOf35BD02fqjvThiht6BD9289/qXPFxeqVbNOE++/Ty6+9rurq6laZPQEAAEDbxIxFE+x2u/bt26fQ0FAFBARIkkpKSjR16lRNmzZNtbW1euONNzRmzBg999xzio+P11tvvdXkWLt379b58+c1f/58TZ8+XRkZGaqtrVV1dbVWrFihSZMm6bnnnlNoaKg2btyo6OhozZs3T5I0b968rwwVDodDy5Yt04ABA3TXXXe528+ePavNmzfrqaee0s9+9jMdOXJEhYWFiouLk9VqVVFRkSQpLy9Pqampzfo+is+fV7It0f3++Wd/ozvThkiS9uzdp+CgIMXGXLsrlM2WKKfTqQsXLjVrbAAAAHQMzFjcYNGiRZIkPz8/xcTEaPbs2e5jycnJ6tGjh6Rrv5RHRkZq4MCBkqSxY8dq3759unTpkrp169ZozCNHjmjixIny9/dXfHy8YmJiVFRUJLvdrpiYGPXu3VuSdM899ygvL6/ZtW7btk319fVKSkpq1P7RRx9p6NCh6tq1qyRp6NCh+vTTT2Wz2ZSamqrc3FwlJycrLy9PU6ZMada1GhwN8vP1u6l94+atenfDZs3995+5A5if37V+doe92Z8FAAAA7R/B4gYLFixQaGjoLftdvXpVnTt3dr/39vZWaGioysvLbwoWlZWVWrFihby8vCRd2z8xYMAAORwOhYSEuPsFBQVp6NChza41KipKkydP1ssvv6xevXq5l0JVVFQoPz9fBw4ckHRtT8j1mYl+/frpnXfe0ciRI1VdXa2ePXs261phncN08dLNMxCbtmzXg5N+oGTbl+Hm0sVr/cLDO9/UHwAAAB0XweI2hIWFKScnx/3e6XSqoqKiyTtIBQcHa9q0aYqPj2/UnpOToxMnTrjf2+12Xbhw4aZ+X+X6rMTkyZO1evVq/eIXv1BgYKBCQkI0btw4jRkz5qZzEhMT5XA4tHPnTvXp06fJO041ZUC/VO0/cEgT7r/PHZBcLpcemzVTvXslN+qbtf+gIiK6KOqfAhYAAAA6NvZY3IZevXrpypUr+uSTT2QYhj788EMFBQU1uR9i4MCB2r59u2pqalRTU6MNGzaooqJCNptNn3/+uQoLC+V0OrVjxw5lZWVJknx8fOTt7a3S0lI5nc6vrWXw4MFKSkrSmjVrJEkDBgxwL8tyOBzau3ev8vPzJV2bWUlJSVF2dnaz91dI0vh7x+nipRJt2LTV3eZ0uvTa63/Tmc/OutsKC4u084PdmviD8c0eGwAAAB0DMxa3wWq16oknntC6deuUkZGhmJgYPfbYY032HTNmjDZv3qzFixfL6XQqLS3Nvdzq0Ucf1bvvvquysjL16tVL06dPl3QtANx11116+eWX9fTTT99yFuOHP/yh/vznP2vfvn1KT0/XuHHj9MYbb6i6ulo2m02DBw929+3Xr5+OHj3q3tvRHN26ddWsh2fozeUrZXfYNXnSBPn4+GjZktfcfY58dFRL3nhLdwzsr1Ejhjd7bAAAAHQMXiVXSl2GYcgwDDkNQ4bTkMPRIIfdLsMwVFdfr7Qhg289EtwWLVqkJ554wv0Au7akqKhIH374oR5//PFvfO7erP1avnK1/P39NWjgAIWHh6umtkYnTuTp7LlijR75PT0yc7osFksrVA4AAIC2jBmLFlZRUaGamhr5+/t7upSb1NTU6IMPPtCgQYNu6/zvDR+m/v1S9cHuPfo0L18n8wvUyd9ftqREPfbow+qZ0Lz9IQAAAOh4CBYtqKKiQn/4wx/Uv39/98Py2pJFixYpJSXFfZvc2xEaGqIHJ03Qg5MmtGBlAAAAaO9YCgUAAADANO4KBQAAAMA0ggUAAAAA0wgWAAAAAExj8/Y/1NfbVVpWruqaWk+XAqANCwzwV5fwMFmtfp4uBQCANoVgoWuh4mzxBUVGhKt7dFdPlwN85xWc+kzJSW3z9sXlFVU6W3xBcbHRhAsAAG7AUihJpWXliowIV1hosKdLAdDGhYUGKzIiXKVl5Z4uBQCANoVgIam6ppZQAaDZwkKDWTYJAMA/IVgAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANN8PF0AAJi1du1aORwO9/tp06bJx8dHdXV1Wr9+vbvdarVqypQpkqTs7GwVFha6j6Wlpclms317RQMA0MEQLAC0e/X19bLb7Te1u1wu1dXVNXlOQ0NDo2OGYbRafQAAfBcQLAC0S5WVlaqpqZF0LUDc6NKlS7JYLKqvr2/U7nQ6dfHiRUlyn3tdRUWF+1i3bt3k5eXVWqUDANAhESwAtEs5OTnKz89v8tj27dubbLfb7dqyZUuTx3Jzc5WbmytJmjlzpqxWa8sUCgDAdwSbtwF0eMHBwfLx4e8oAAC0JoIFgA5v1KhRioiI8HQZAAB0aAQLAAAAAKaxNgBAh2SxWNyvvby8ZLFY3G1Op/OmDd8AAMAcggWADumBBx5wb8C2Wq0aPXq0+5ayhw4dUlFRkSfLAwCgwyFYAOiQbnww3sSJE3X48GH37WQBAEDLY48FAAAAANMIFgAAAABMYykUgA5vy5Ytcjqdni4DAIAOjWABoF3q3bu3unfv3ipj+/r6tsq4AAB0ZAQLAO1SREQED70DAKANYY8FAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWAhKTDAX+UVVZ4uA0A7UV5RpcAAf0+XAQBAm0KwkNQlPEyXr5QRLgDcUnlFlS5fKVOX8DBPlwIAQJvCk7clWa1+iouNVmlZuS5fKfN0OQAkFZz6zNMlNCkwwF9xsdGyWv08XQoAAG0KweIfrFY/dY/u6ukyAAAAgHaJpVAAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA03w8XQDQXPX19TqZX6DLV0rlY7EoOjpKtqREeXuTjwEAADyNYIE2zzCc2rBps7Zs2yG73aHQ0BAZDYaqrl5V57AwTZs6Wel33+XpMgEAAL7TCBZo0xwOh/70f19V0ekz+uGDkzRi+DAFBgZKkq5cKdXW7e/rr397U+fOndf0aVM8XC0AAMB3F8FCktPp1Lx5825qnzFjhgYPHnxbY77yyiuaMGGCEhISTFb37Tp48KASExMVGRl507Ft27apqqpKU6dOlSSVl5fr5Zdf1sSJE3XHHXe0Sj1vvb1an509p9/O+6ViYrpr/bsbtevD3fLz9dPkByfqkZnT1SvZptde/5uio7pp5IjhrVIHAAAAvh7B4gYLFixQaGhoi4z14IMPqlu3bi0y1rfp4MGDCgkJaTJY3Ki+vl5Lly7V3Xff3Wqh4uy5Yu3N2q///dN/UWxsjHbvydKmLds0+YGJqqqq0tJlyxXdrZvuTBuiM5+d1Zp1/6O770qTn59fq9QDAACAr0awuIWLFy9q2bJlGjhwoA4ePKigoCA9/PDDio6O1sqVKxUbG6uRI0dKkpYvX67ExEQNHz5cGRkZmjx5spKSkrRw4UL16dNHubm5+vnPf66wsDBt3rxZhw8fVqdOnTRixAh973vfkyTt27dPp06dktPpVGFhoeLi4jRr1ix16tRJe/bsUUFBgQzD0JkzZ5SQkKDx48drzZo1KisrU0pKimbMmCFvb285nU5t2rRJ2dnZCgoK0qRJk9S3b19J0sKFC/X9739fWVlZstvteuCBBzRo0CC98MILKi8v1+rVq3XfffcpPT29ye/E6XTq7bffVnR0tMaNG+duv3r1qt555x2dPn1a0dHRmj59usLCwvT888/rqaeeUo8ePWS327Vw4UL96le/UufOnb/2uz946LCiunXT4EEDJUmnPzurcWNHa9KE8ZKkE5/m6UTeSdlsifrB+Hu1dfv7yj2Rp0F3DDD3QwcAAMA3xu10mqGsrEyhoaFasGCBkpKSlJmZKUnq37+/Tpw4IUkyDEMFBQXq169fk2P4+/tr/vz5Cg8P165du3Tu3DnNnTtXc+bM0Z49e/Tpp5+6+548eVKjRo3Sb37zG9XV1eno0aPuY+fOndOECRM0f/58Xb16VW+//bZmz56tuXPn6ty5c8rLy5Mk7d69W+fPn9f8+fM1ffp0ZWRkqLa21j3O+fPn9cwzz+jBBx/Ue++9J0maN2+eunfvrhkzZnxlqJCk9957T7W1tZo2bVqj9oyMDHXt2lXPPfec+vfvr4yMDFksFneokqSCggJFRkbeMlRIUvH580pOTnK/n/3IQ3rox9eWYeUXFKrk8mX16Z0sSQoKClRUt64qPv/5LccFAABAyyNY3GDRokWaO3eu5s6dqz/96U/u9qCgIKWnp8vX11d9+vRRWVmZJKlPnz76/PPPVVNTo9OnT6tr165fuZRq2LBhCggIkJeXlz766CONGzdOgYGBioyM1IgRI3T48GF33759+yohIUH+/v5KTExUaWmp+5jNZlP37t0VFBSk5ORkpaSkqEuXLgoJCVF8fLy775EjRzRmzBj5+/srPj5eMTExKioqco8zduxYWa1WpaSkqLq6Wna7vVnf0YkTJ5SdnS1JjW7zWl1drYKCAt17773y8fHR8OHDVVxcrLq6ukYBLC8vT6mpqc26lsPRID9f35va807m6w//9d+aMvkB9Uq2udv9/PzkcDTvcwAAAKBlsRTqBs3ZY+Ht7S3DMCRJvr6+Sk5OVl5enoqLizVgQPOW4Fy9erXRX+w7d+6sY8eONdnXy8vLfb2mavnnvk6nU5JUWVmpFStWyMvLS9K1GZWm6rs+xlddo6n+zzzzjFasWKH3339f9957r/t6LpdLv//97xv1r6qqUu/evbV69WqVlZXp5MmTmj17drOuFd45TBcvXbqpffuOnerdK1nj7/1yGZbT6dTly1cUHh7erLEBAADQsggWJg0YMEDHjx/XhQsX9NRTTzXrnNDQUJWXl7s3SH/xxRcKCgpq0bqCg4M1bdo0xcfHt+i4ffr0UZcuXTRz5ky99NJLSk5OVs+ePRUcHCwfHx8tXLiwyQfW9erVSzt37pQkxcTENOta/ful6q9L31RZ2RcKD/8yiI0bO/qm7+vY8RzV1NYqNaWPiU8HAACA28VSKJP69u2rgoICderUqVn7BiRp8ODByszMVG1trb744gtlZWVp0KBBLVrXwIEDtX37dtXU1KimpkYbNmxQRUXFLc/z8/NTaWnpLWcwIiMjNXHiRK1atUq1tbUKCgpSXFyctm7dKsMwVFJSovXr18vlckmS+vXrp+zsbKWkpDT7MwwdMlhduoTrjTdXuGdiJGn9uxu1b/9B9/uqq1e1cvUa3ZU2VJEREc0eHwAAAC2HGYsbLFq0qNH7KVOm3PI5FH5+frLZbOrRo0ezrzNy5EhVV1dr8eLFcrlcGjlypAYOHHg7JX+lMWPGaPPmzVq8eLGcTqfS0tKadSvdO++8U+vXr5dhGBo1atQt+548eVJr167VrFmzNGPGDK1bt07PP/+8/P39NX78ePdSrJSUFHl7ezd7f4UkWSze+um/zNF/vLhY//3ya3ry8UcVEhysBfN+6e5TXHxer/6/JbJ4e2vWw9ObPTYAAABallfJlVKXYRgyDENOw5DhNORwNMhht8swDNXV1yttyO09JA64zuFw6MUXX9T8+fNlsVi+0bmnz3yml197XVevXtUdAwcoJjpaDYah06fP6JMTn8qWlKh/+9enFRoa0krVAwAA4FYIFmh1DodDH3zwgSoqKtxP7b6dMfZm7dfHxz/RldIyWSzeio6K0p1pQzTojgHumREAAAB4BsECrW7JkiWqqqrSk08+qZAQZhUAAAA6IvZYoNXNmTPH0yUAAACglXFXKAAAAACmESwAAAAAmEawAAAAAGAaeyz+ob7ertKyclXX1Hq6FABtWGCAv7qEh8lq9fN0KQAAtCkEC10LFWeLLygyIlzdo7t6uhzgO6/g1GdKTor3dBlNKq+o0tniC4qLjSZcAABwA5ZCSSotK1dkRLjCQoM9XQqANi4sNFiREeEqLSv3dCkAALQpBAtJ1TW1hAoAzRYWGsyySQAA/gnBAgAAAIBpBAsAAAAAphEsAAAAAJhGsAAAAABgGsECAAAAgGkECwAAAACmESwAAAAAmEawAAAAAGAawQIAAACAaQQLAAAAAKYRLAAAAACY5uPpAgDArLVr18rhcLjfT5s2TT4+Pqqrq9P69evd7VarVVOmTJEkZWdnq7Cw0H0sLS1NNpvt2ysaAIAOhmABoN2rr6+X3W6/qd3lcqmurq7JcxoaGhodMwyj1eoDAOC7gGABoF2qrKxUTU2NpGsB4kaXLl2SxWJRfX19o3an06mLFy9Kkvvc6yoqKtzHunXrJi8vr9YqHQCADolgAaBdysnJUX5+fpPHtm/f3mS73W7Xli1bmjyWm5ur3NxcSdLMmTNltVpbplAAAL4j2LwNoMMLDg6Wjw9/RwEAoDURLAB0eKNGjVJERISnywAAoEMjWAAAAAAwjbUBADoki8Xifu3l5SWLxeJuczqdN234BgAA5hAsAHRIDzzwgHsDttVq1ejRo923lD106JCKioo8WR4AAB0OwQJAh3Tjg/EmTpyow4cPu28nCwAAWh57LAAAAACYRrAAAAAAYBpLoQB0eFu2bJHT6fR0GQAAdGgECwDtUu/evdW9e/dWGdvX17dVxgUAoCMjWABolyIiInjoHQAAbQh7LAAAAACYRrAAAAAAYBrBAgAAAIBpBAsAVmTd6QAAATNJREFUAAAAphEsAAAAAJhGsAAAAABgGsECAAAAgGkEC0mBAf4qr6jydBkA2onyiioFBvh7ugwAANoUgoWkLuFhunyljHAB4JbKK6p0+UqZuoSHeboUAADaFJ68Lclq9VNcbLRKy8p1+UqZp8sBIKng1GeeLqFJgQH+iouNltXq5+lSAABoUwgW/2C1+ql7dFdPlwEAAAC0SyyFAgAAAGAawQIAAACAaQQLAAAAAKYRLAAAAACYRrAAAAAAYBrBAgAAAIBpBAsAAAAAphEsAAAAAJhGsAAAAABgGsECAAAAgGkECwAAAACmESwAAAAAmEawAAAAAGAawQIAAACAaQQLAAAAAKYRLAAAAACYRrAAAAAAYBrBAgAAAIBpBAsAAAAAphEsAAAAAJhGsAAAAABg2v8Heg6qijPgOdIAAAAASUVORK5CYII=)

### Project Key (Required)

In this text box you need to provide the key for the **project** your metric events pertain to. You can find it under Environments on the Projects tab on your LaunchDarkly Account settings page.

### Environment Key (Required)

In this text box you need to provide the key for the **environment** your metric events pertain to. You can find it under Environments on the Projects tab on your LaunchDarkly Account settings page.

## Authorization

![Authorization section with a single empty text input field labeled Access Token.](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAuUAAACDCAYAAAAqCGWQAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABUtSURBVHic7d17cFRlnsbxp9OdNLk3uRIISSABhkDIoMRwWS4RcIRBQATB8YLrouWKVWs5VSg7IzBbs7Wu5U7V6q7FzGJhMeIYhhGMGBAFgnInXEIIkCCEYEK4hnTu6eR07x9MjkTCACN4DPl+qlJJ3tPnfX+nwx9PXn7nxHbZXePzer2SJK/XK5/PJ8MwZBiGvIYhw2uopaVVLR6PDMNQU3OzMu69RwAAAABuDz+rCwAAAAC6OkI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFC+S3Kzc3VggULdODAge8919mzZ7VgwQJt2rTpNlT2rRUrVuidd965rXO2+eabb/Taa6+puLj4jswPAADQFRHKb9GhQ4ckSQcPHrzlc//whz9o0aJFt7uka4SFhcnlct2Wub5bc0BAgLp37y6n03lb5gcAAIDksLqAW3Xq1CkdP35cEydO7PD4559/rn79+ikpKem2r3369GlVVVUpKipKJSUlamxsVGBg4G1f5+/l9Xrl5+en6dOn37E1YmNj9fLLL9+x+QEAALqiThfKN23apOLiYhmGoQcffLDdsfXr12vLli365ptv9Mwzz9z2tQsKCmS32zVt2jS9++67Onz4sDIyMszjixYtUkJCgubNmydJ2rJli9avX6+XXnpJS5cuVVNTkyRpwYIFeuGFF9StWzdJktvt1rJly1RWVqaYmBjNmTNH0dHRkqSGhgbl5OToyJEjkqRBgwZp2rRp6tatm86fP68333xTQ4cOVUVFhVwul+bNm6e33npL9fX1WrhwofLz87Vq1aprrmX69OkaOXKk6urq9Mknn+jYsWOSpP79++vhhx9WUFCQFi1adE3NjY2NWr58uR599FENGzZMklRZWamcnBydPn1aQUFBGj58uMaPHy9J2rFjh9auXaupU6dq586dcrvd6tevn+bMmcNuOwAAwF91uvaVJ598UklJSdq8ebM2bNhgjrcF8qSkJD3xxBN3ZO1Dhw4pJSVFAwYMUFhY2C21sDz22GOKi4uTv7+/5s6dq5iYGPPYnj17FB0drYyMDJWXl2vdunXmsffee08FBQUaPny4MjIydODAAWVnZ7eb+8CBA4qMjNTAgQOvWTchIUHTp0/X9OnTNW3aNAUFBclut6tPnz6SpJUrV6qoqEjjx4/XuHHjVFhYqI8//viGNbepq6vT0qVLdfHiRU2YMEG9e/fWZ599pq+++qrd6zZv3qwhQ4YoJSVFRUVF2rVr102/dwAAAHe7TrdTHhAQoHnz5mnZsmXavHmzpCttG3l5eUpKStK8efMUEBBw29ctLS2V2+3WhAkTJEmpqanas2eP6uvrFRwcfMPzBw4cqK+++kqXL1/WoEGDJEk1NTWSpNGjR+vnP/+5pCstMuXl5ZKksrIynTp1ShMnTjTbdfz8/LR161adO3dONptNkjRs2DA9+uijHa4bExNjhult27apoaFBkydPVlxcnCRpzJgxCgoKUmJioiSppKREJ0+evG7N37Vnzx41NjZq7ty56tu3ryTp7bff1ubNmzV69GjzdTNnzlRqaqpaWlr061//2rxGAAAAdMKdcunbYN62Y36nA7n07Y2dsbGxunTpkuLj4+X1es0bP7+PtjaWtq+bm5slSWfOnJEkc1dbklJSUiRJFRUV5lh4ePgN1zh//rxyc3OVlJSksWPHmuM9evTQrl279Prrr+u1115TaWmpuf7NqKyslM1ma9fDn5ycrPr6elVXV5tjQUFBkiR/f3/Z7XZ5PJ6bXgMAAOBu1+l2yttcvWMu6Y4Gcq/Xq8LCQkm65lGDBQUFGjFixB1Z1+G48uNp2xFvq+XqzzfD6/Xqww8/lN1u15w5c9rNt3z5cjU2NmrGjBmKiorS6tWrVVlZedNz2+32dvNJks/na/cZAAAAf1unDeXSt8G87es75cSJE6qrq9OIESPMnWpJ2rt3r4qLi1VTU6OwsDA5nU7zxkhJ1+w422y2WwqqsbGxkq60ziQnJ0u68vQZ6coO98364osvVF5erlmzZikiIsIcr6ur09mzZzVy5EizH90wjFuquUePHvJ6vSorKzN39EtLSxUYGHhTO/gAAADo5KFcurNhvE1b68q4cePUvXt3c9xms+nYsWMqKCjQ6NGjFR8fryNHjmjdunVqbW3V7t27280TGhqq5uZm5ebmKjMz84brJiQkKDk5WZs2bZLH45HP59O2bdvUt29fxcfH6/z58zeco7y8XJs3b1ZYWJgMwzBvsIyLi1Pv3r0VHBysgwcPyuVyqaKiQqdPn273nt6o5szMTOXl5emDDz7QyJEjzTkmTpwoP79O2R0FAADwgyM13YBhGDp8+LB69OjRLpBLVx4f6HA4VFBQIEmaMmWKEhIStGvXLpWXl18TYseOHauoqCjt3LnTvMnzRh5//HENHjxYO3fu1J49e5Senq65c+fedP2lpaXyer2qqanRRx99ZH4UFBTIz89PTz31lMLCwrRp0ya1trZq8ODB8ng8qq+vv6maAwMDNW/ePHXv3l0bN25UWVmZHnjgAfOGWAAAANyY7bK7xnd1n7LP55NhGDIMQ17DkOE11NLSqhaPR4ZhqKm5WRn33mNx2QAAAMDdg51yAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBiDqsLQNfl9Xp14mSpzlSeVWtrq6IiIzVgQD91czqtLg0AAOAHRSiHJXbvzdeqP6/RpaoqhYQEy+FwyO2ukb+/v342cbymPTRZDgf/PAEAQNdA6unilixZohdeeEExMTE/2Jp/WZOjdbkbNG7saE362QTFREdLkuobGrR9+y6tyVmnkuNf65cvvaiAgIAfrC4AAACrEMpv4J133lFcXJwefvhhq0u5ri1btmj9+vUdHnvllVcUGRn5A1d0fTt27da63A169p/mauTwTO3bf0Bv/u5tNTQ2aOTwTM2eNUODB6fqP974L723YqWem/ePVpcMAABwxxHK/4bLly/r3LlzunDhgqZOnSq73W51SR3KyspSVlaWJGnZsmUaMmSI7rvvPourulZra6tW/XmNJo7P0sjhmao8e1bv/P5djRyRqV4947T243UKCwvVlMkP6tlnntbv/vt/NGF8lvr2SbK6dAAAgDuKUP437N+/X2lpaTp37pyOHTumQYMGmceqqqq0atUqlZeXKyYmRo888oh69eolSdq6dau2bt0qr9erYcOGacqUKZKkuro6rVq1SqWlpYqLi9Ps2bPNXezdu3fr888/V0tLi9LS0jRjxgz5+fmptrZW2dnZKisrk8vl0owZM9SnT59buo6ysjKtWbNGly5dUmJiombOnCmXy9XuNT6fT++9955CQ0M1c+ZMeb1effrpp9q7d69CQkL00EMPaeDAgZKutLzcf//92r59uzwej6ZOnaqhQ4fesI6jxSVy19Ro8qQHJEknTpQquW8fPTP3CdlsNl28eElHjhZryuQHNSRtkHr3jtfuPfmEcgAAcNfjkYh/w4EDB5SWlqYhQ4Zo//797Y6tXLlSycnJWrJkiUaPHq0VK1ZIkoqLi7V9+3bNnz9fCxYs0KlTp5Sfny9Jys7OVkxMjBYvXqy0tDRlZ2dLktxutz799FPNnz9fr776qs6cOWOe89lnnykmJka/+c1vlJWVpQ8++OCWrqGxsVHLly9XVlaWFi9erMTERLPWq+Xk5Mjj8ZhtOl9++aUqKiq0cOFCzZ49W9nZ2WpsbDRfX1FRoZdfflnTpk1TTk7OTdVSXl6h6KgoucLDJUn/MGqE/vWVX8pms6m62q3CoiMa+JP+5uv7JfdVecWZW7peAACAzohQfh3l5eWqra1VSkqK0tLSdPToUTOUut1uVVZW6v7775fD4dDQoUM1ZswYtbS0qLCwUPfdd58iIyMVFBSk6dOnKzw8XPX19Tp+/LgeeOABORwOjRo1SuXl5WpqapLdbpfP59P58+fldDr13HPPaciQIZIkf39/ud1u1dXV6Z577tGLL754S9dx7NgxRUdHKz09XQ6HQ+PHjzfbctrs3r1bxcXFeuqpp8wWnX379ikrK0uBgYFKTExUr169dPLkSfOc8ePHy+l0KjU1VfX19fJ4PDespaWlVQEB/teM19TW6t/+/T/Vq2ecpkx+0BwPCAhQS0vLLV0vAABAZ0T7ynXs379fqampstvtcrlciouL06FDh5SZmamGhgYFBwe36zEfNWqUJKm+vl4JCQnmeHx8vCSpsrJSPp9Pv/3tb9utU1tbq+joaM2dO1dffPGF/vSnPyk9Pd1seZk8ebI2bNigt956S2FhYZo0aZLC/7rTfDPq6urUvXt383s/Pz+Fh4erurpasbGxkqRdu3YpKCio3Xk1NTV6//33ZbPZJEmGYZi/KFzNz8/PPH4jEd1dunjxkgzDaPfe7dq1V03NTXr+2WfM9STp7LnziriqdgAAgLsVobwDXq9XBw8eVF1dnfbt22eO2+12ZWZmKiwsTPX19e3C5cmTJ5WQkKDw8HDV1NSY51RXV8vj8Sg0NFQOh0NLliwxg2yburo6RURE6Pnnn1dDQ4P++Mc/aufOnRozZowuXLigSZMm6aGHHlJRUZHef/99LV68+Jo5rsflcqmwsLDdtbndboWEhJhjTz75pPLz85WTk6PZs2dLkkJDQzVr1iwlJibe+ht4HYNSB6rZ49G+/Qd1X8a95njqwAHq2TOu3eMP3e4aHTl6VE89/thtWx8AAODHivaVDpSUlMhms+n111/XG2+8oTfeeEO/+tWvdPr0aVVVVSk4OFgJCQnKy8uT1+vVkSNH9OGHH8rPz09paWnau3evqqqq1NTUpOzsbJ08eVIhISFKSEjQhg0bZBiGzp8/rzVr1sjn8+nMmTNaunSp3G63nE6nunXrptbWVknSmjVrtHXrVvl8PoWFham1tVU+n++mr6V///66ePGiDh8+LMMwtHXrVoWEhJg3pUpSRESEZsyYoZKSEhUVFUmS0tPTtXHjRjU0NKihoUGffPKJ3G7393pfu3d3adTI4fpw1V/kdn/7i8vuvfu0+qO15vc+n0/LV6xUWFiYhmdmfK81AQAAOgP7qwsXLmkLeVd/vvrD6/XKaxjy+XxqNQz16hlnZc133MaNG5WSkqJ+/fqZY926dVNFRYXq6+vVt29fpaSkaMeOHfr4449VWVmpp59+WiEhIYqIiJDdbtfq1auVl5en9PR0jR07VpLUr18/5efna+3atSosLNSwYcMUFxenyMhI+Xw+ZWdnKy8vT9HR0Zo0aZLsdrv69++vvLw85eTk6Pjx43rkkUfMtpOO7N+/X7GxsWbodjgcSk5OVm5urnJzc+XxePT444+b7Sp5eXnKyMiQy+VSTEyMVq9erWHDhiklJUWVlZX66KOPtG3bNsXHxystLa3dOcHBwfJ6vdq0aZOysrLk739tv/h3DfxJf23fsVtfbt+hAf37KTw8TKkDB2jc2NGSpNq6Ov1+2XIdLjqql/9lviIjI/6+HyIAAEAnYrvsrvF5vV5JV1obfD6fDMOQYRjyGoYMr6GWlla1eDwyDENNzc3KuPcei8tGZ1ZbW6f/Xfp/Ki45rtSBP1Fy3yQ5HP46U1mpgwWFCgoM1Px/flbJfW/t0Y8AAACdFaEcljlYcEi79+7TmcqzMlpbFRkZoSFpgzV61MgOn9ICAABwt+JGT1jmp+lD9NP0a5/oAgAA0NVwoycAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMW70vI7mZo8uVVWrvqHR6lIA/IgFBwUqMsIlpzPgxi8GAOA6COUdaG726HR5paKjItQzLsbqcoAu7/iJMvVLTrS6jA5Vu2t1urxSCfFxBHMAwN+N9pUOXKqqVnRUhFzhoVaXAuBHzhUequioCF2qqra6FABAJ0Yo70B9QyOBHMBNc4WH0uoGAPheCOUAAACAxQjlAAAAgMUI5QAAAIDFCOUAAACAxQjlAAAAgMUI5QAAAIDFCOUAAACAxQjlAAAAgMUI5QAAAIDFCOUAAACAxQjlAAAAgMUcVhcAALfb6tWr1dLSYn4/a9YsORwONTU1ac2aNea40+nUjBkzJEl79+7V119/bR7LyMhQSkrKD1c0AKBLI5QDuOs0NzfL4/FcM+7z+dTU1NThOa2tre2OGYZxx+oDAOC7COUA7go1NTVqaGiQdCV8X+3cuXOy2+1qbm5uN+71enX27FlJMs9t43a7zWOxsbGy2Wx3qnQAAAjlAO4OhYWFKikp6fDYxo0bOxz3eDxav359h8eKiopUVFQkSfrFL34hp9N5ewoFAKAD3OgJoMsJDQ2Vw8GeBADgx4NQDqDLGTt2rKKioqwuAwAAE6EcAAAAsBj/fwugS7Db7ebXNptNdrvdHPN6vdfcHAoAwA+JUA6gS5g6dap5s6bT6dS4cePMxx7u2bNHJ0+etLI8AEAXRygH0CVc/UeDpkyZovz8fPORhwAAWI2ecgAAAMBihHIAAADAYrSvAOhy1q9fL6/Xa3UZAACYCOUA7goDBgxQz54978jc/v7+d2ReAADaEMoB3BWioqL4g0AAgE6LnnIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoTyDgQHBaraXWt1GQA6iWp3rYKDAq0uAwDQiRHKOxAZ4dKFi1UEcwA3VO2u1YWLVYqMcFldCgCgE+MvenbA6QxQQnycLlVV68LFKqvLASDp+Ikyq0voUHBQoBLi4+R0BlhdCgCgEyOUX4fTGaCecTFWlwEAAIAugPYVAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYv8PvbgynTL/D5sAAAAASUVORK5CYII=)

### Access Token (Required)

In this text box you need to provide the access token to be used for authorizing the requests to the LaunchDarkly API. This can be either a personal or service token. The access token must have a role that allows the `importEventData` environment action. It is strongly recommended to use a dedicated access token with this permission.

## Metric Options

![Metric Options section with a single empty text input field labeled Event property for Metric Value.](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAw0AAAB+CAYAAABmv/0SAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABvoSURBVHic7d1/dNT1ne/xJ8lMwjdxAhlwgASYYAYY0hlkogQbe4MFa/RCbVzBI9xKT3Gvei/bleuyXm2Xdbt2+2OX3eq23mt7r7QFhfaCGhWEqIkSa5SgRJhZGEoiTDEBpjAxGchA5kfuH0kgQDKEIobA63EOB/L9+f5+J5zzeX0/n893hjS3tHYkEgkAOjo6SCQSJOJx4okE8XicWCxGPB4n2t5OLB6jvT3KDZ5piIiIiIjI1SFloAsQEREREZHLm0KDiIiIiIgkpdAgIiIiIiJJKTSIiIiIiEhSCg0iIiIiIpKUQoOIiIiIiCSl0CAiIiIiIkkpNIiIiIiISFIKDSIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk0iIiIiIhIUgoNIiIiIiKSlEKDiIiIiIgkpdAgIiIiIiJJmQa6gEFh/wZ+/Gw1IXIpXfows0efuTr03i9Z8Uo9scwiFi+fh/PzimL7K3j6N3Xkzn+YeQXGn3GAGMHtm9n8ex8Nf4pgGpbLlBmzKL3ZgeViarzoukRERERkMFFPw4VINOLzNZ21MITft5/YBR3Hx9onHuW7z9cl3y/NwJJtxZJuvuBSAYLvruSZ/1dNQyIHT5EHhyXE9tdW8otX/EQuptaLrEtEREREBpdB19MQ2V9JxT4HpV+1c+4z7giBtyuon1DK7LzP/wm4Kd3EYZ+Xplk55HTHrWYfvkAMU7rpwoJDf+SUsPivS/68fY/XsbmqnujYUv7qodnkmADCONc8xdqtFVRPd1I6dgDqEhEREZFBZ5CFhhDe6hpqdlXTGL2Pxbc5egSHCPVvrGR1VQBTQQ6evCKsn+u5TeRNdNDo9+I9VEpOTldFPj/7Ux04JzTiO9Bz+zD1b2+g4gM/jW0mrBM83P6N23GdrOTpf6+kMQHsXMt3n/Cz6IkF2Let5EcvB3FOt9Ps82K6ZSlLRlfzxHPbyb93OYsKDSBG09YNbHi3jv0hyMpxUnxHGSX5vcSnvV4ajptwzCnuCgwAFjzFHip2VuPf1UTp2BzCW1fyo5dDOGfkEt7lpbE9i1zXLObfVYQtWNFrra6968+pK7hzMxuq6ggcjWAa4aTotrmUFliBGL41f8+qgJvSYvD93svh9izyispYMMeJBSAWpG5jOZUf7ycUM07fqxGD7NdTRERE5Ao1yIYnWSm6dxGl+SYCVatZ+UZ91zCbHoEhv5T77/28A0Mn0wQnzvQQ3p3dQ5RC+Lz7YYILR8aZ2za9tYqVFfWYnSXcPstN1sFq1qyupGmYhznfnI0jHUx5JSxaOBN796eQCOHzBcl2FeHOOTcIhHeu47mXa2m1Tef2O4rJafey4fl11LacW2v4SDORFCs221nHGWHDlgKHj4RO94wkgjQ0GnjmLKTsRivBj9azqiJALDtJrT1E9pSzck01jWlTKLltNlPSAlQ+v5INDT36XprrqPkPM66vzKZwRIT6d8vZsv/0vVq7tRVbSRnz/7OHrMZq1qzeQlMi6cchIiIiIl+QwfcoN83O7G8tgt+soqJqNSsT85nJFta90xUYvjWbnLRLc+qYyYF7shnvLi9Nt+WQ0+zD+yk4/sKJ4d/QY0M/Ne8HMBfMY8GcQgzAlRZkxWtevEdLKS3IZVsKkJWLc3IOJiAMgIHnGw+wYFpXQ39Pz7OH8W71ErYWsWDhXBwmKLHDMy/U8ccDMYqGnflRRk5GATPG2fci3YwpFYhFTy9LseApLaN4IjAthxMHVrBhp5f9d8zF0UutZwrjfW87oUwPi749D5cBFFppW7GWbe95mZ3v7jqvi7nfnofHgFhOM7uf207TwTDkGRw+EgLDTeFNRbgMsFut+IJZmBIMulgrIiIiciUafKEBzgoOq1gFWCZe2sAAQMxE/lQ3xk4v2z+dzdB9XhrNTuZPMYj6emwXaqQxAhHfep78u/U9VhiEWpMcP8WEkdnHXIzEYYLBGKbcHHK7P7XxpSx5vLTXzc3pZiBKpP2sFSejxOKAqa9JzFZyxhjwaZDQSSA9Sb1ddTUdjmHKzcfeXXqmA0eOCV+wkWCiKzSkmDF3HcuUPhQDiMaigAXn9CJse2pYsyKIw5mPY3Ih02/OwVBgEBEREbksDM7QAGcEhxqKL31gAKKxGIbDxRRjO/4d2zECjZgnzseZGcPbcyhN17+thfNYcNOo08tTwHLtRRRwAY3o7JHZmBJ+gsEIjO8RRI4GCSZg1Ehrnx9+rPtaLnB40OnjRTv37ef+xuQylj7iYvtHPvx/8LF5WzVbXPNYsujSDDMTERERkQszuJ/lptmZ/a1lLPsCAsPpczpwFRgEt2+k+lMzTpcTAzqf3nezjsKWDq1Hwhjj7djz7NjzrJhNVqwG/FlZLWUUtpEmYsEmGrunCgRrWb9qFZUN5763ych348yMUf9BDU2nVkfwfeAjhA1nQc4Z28eiXRslmmg8EAaLjVH9qTVlFDmjTMQO/pFAd69GpInAoRgmW+f8iaQSQeo2b2BzQwZFt5ax6L8vY/GXrYT9H+PvZa6GiIiIiHzxBm9PQ7c0o5dXr15KJhxT3Vi31RAyPLgnG3TPSDhdk5PiGbl4qypZ+esw0ydaieytoabRzsJlC3ClG1gyIRbYxua3zRTNdPXjGiy4Z7ipXFNL+RqDognQ+FENdS35zPtGLx+jxcPtt2yjYVMFz/2vIJ5JVmJNXmr9rVhnLKKk5+tWE2G2v74Gy1EHpgPbqGwyYZvl7pz0nDi3VtvZdd1USNWqWtb9xsTMAish3xbqIlaKb3af/7pSDAjVUf1+AyeOl+C0hPHuawWLE1vmeW+KiIiIiHwBBndPwwAxTXDhHAbGZDfOXlvFJuy3LWbxHW4sh+uo3FSJr83O7Hvmdk4UTrFTNMuDLbafmg/8hPr5BQ+WqfO5/65Csg7VsHlTDYE0N2Xfnk/RsN63t828jwfvKcFOgG3vbcEbysJ9xyIeuMt5ZmM+xSBvQgb1v6+gcneE3BllLLrV3rXu/LUaBWUsvreE3LCXzRsr8UZyKVn4AHMn9ifOWfDcdT9lUw32v7OONesracwspOy+UhyDP9KKiIiIXBGGNLe0diQSnYPPOzo6SCQSJOJx4okE8XicWCxGPB4n2t5OLB6jvT3KDZ5pA1y2fF46v6ehkaL7l1M2caCrEREREZHLkXoaREREREQkKYUGERERERFJSsOTREREREQkKfU0iIiIiIhIUgoNIiIiIiKSlEKDiIiIiIgkpdAgIiIiIiJJKTSIiIiIiEhSCg0iIiIiIpKUQoOIiIiIiCSl0CAiIiIiIkkpNIiIiIiISFIKDSIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk0iIiIiIhIUgoNIiIiIiKSlEKDiIiIiIgkpdAgIiIiIiJJKTSIiIiIiEhSCg0iQjQ60BWIiIjI5cw00AXIVSIRou7F1WzYGcJ+12MsKjQGuqLLS7iWlf/WQNHyBbi+iCh/8gg73trM5q1e9h5sJZoAMkbicE6j5PavMTM/6wsoQkRERAYLhYbexHys+v4qfCd7LjThWviPLJr2Od+yo36q9xlMv9HOoGhGx5qoe+8wo272kHMBtyLWUMWGgJ35jy7Babm4e9j0xtM89Y6JuX+zhJIRPddE8K35Mav2OFn0xPka3xECH24jMqEE54jet6h/bQUbjYU8fGtO/woLVvLM/25k+rJFFGWeXhzeupIV2xws+asSbP070iUVbXyP537+O95vGcH1xbdw352jyUyBaPMBdnzwHqt//B7Vtz/I39w9mYyBLlZEREQuCwoNfUlxUPb4AxRnX+LzHPGxpdaGa7CEhuhh6qrq8BRdWGiItrQSG+3BcZGBoZsp0UhdXZCSW3s0w8NeavdG+nmECIHaLQQtfYeGvOL5lKVcQDPf5sF9bTX+PRGKTvWkhNm9O4Dt+rLLIjBwZBvP/uvz7BpxK//zsTu51v87nl39KvVtZq5138lD/+MH3Pnhr/i3Nc/wY5ay/O7rMA90zSIiIjLgUh97/PF/6OjoOLWgo6PjjD+JRKLz73icREeCeDxBzpjRA1jyFyARZEf1Aaw33cC4M1ryMXy//SfKm93cYDdO/xwu5IZx6UQ+rWH9r3/N+g3V7GwawhinneGpEH7/l/zz20c4UVfO6hcr2NrQzugCB2xfxdPlXlqaA+yoO07uTZOwDulxukOVPP3sO3z6SQ2vb6ygsq4Jwz6F3GtSOtf97C3+sPdNXqr8jOtumsTwY/VU/m41L7z0KtV1AU5YHThGpnfWueb7rPMe5ePK13n9zWr2REYyyTGSdIBECN/ra1m1dh0VtQ0cHzaBSTYDEvWU/8tKavbVseHFbQwZEabiNxV80naEvdv9JK5zc+SlH1Decj1FeQYQpub//JC3h9zE9WNONzUjH6/l6df8hP+0l9oPDzGi0I3NHCP44au88PxaXnxjK/4jaYydmIslld6vrcd9CTdspd48hkTgM3JnnF4X+nAjlSdsjDlmJm+mG9sQ4Hg9lb9dzQvrN1G96yjG+EnkZnzChp+u5PeHwjT6PyKQXoAnt/HMa/1SEfEPnuW1Pzm56bpMIELg3fX8Zs16Xq3aSkN4OBMm2jB6fl4YWE7s5c36dIqmjulsbB/fQcWmNtxf/wrjDIgF63h1zQuse7WCau8hzOOmMM6SAu2N1L3fTO5MN7YOP+t/+DuOfKkIewYQqWPlDzbBTR7GmPu4pmv6M6aplff/78/Z1H4zSx+9h4Jjb/CTp7aQuPEeFn0tl9AHr1Jx8DruuusObsz8hI0vf0B06i0UDO/X/xoRERG5gmki9AUx4XQ5CPn9hAFi9fj3jcLltEDEz8Y1NZhnLWH5E0soMdWy7s3AqT0jgUaMWx5k+XcfZHq0hqrtIawzFvHYvYVYxs5mybK5OHr7NMIRsm95kGWPP86DnjAVL9UQ7F7X1kpW0QMsXzoXe0qI2nVr8Y8oZenyH7L0G7nUr19LTXP3xlHCKfks+OtlLF9ahtVXTsWuGADBd9ey4bCDhY/+I4/dm0/TaxuoDXftFg8TGVnK0u8toWTqbB54pAxHuoOyR5Ywe6zBFJedoN9PCOD4bvyH7DgnntlnYkxbwMNzHBiTy3j80QW4DIg1bGZVVSvTvvkYP/zeYqZHq1izqZ5Yr9fWy3251oUnzUddQ/ceQXw7QkzxOHo8GQ9T++J6/LY5LH1iOQ/e0ErVSzUEUxzMfWQJs8daKLz3MRZ/2XrutZ7VLRDbU8HabWZmP7ScHy5bQH7Ty5RvC3M2q8vFqICfhq4Oj8geP43jXLizO2usWV9Bq2sRj39/Ofe7Q1S8Vtt57/qtj2vqz65HanlrF9xYdicFGRBtgZwb7+b+RTO58ct3ct9Xx9G2dw+BOFz71bu5Y+whtrztQ3OkRURERKGhL4l6yn/0KI8+2vnnuz+vJAiYJrmxH/GzOwyxgJ96qwtnNsT21uEdVswslxWTyYqn2AkN9QQTnYczTyyiOM+CycjBcZ2F1s/ObXD2ymLHkWMCTNhmFOM46qe+pWvdMCeeAismE9Ds4+NGBzO/5sRiAkv+bGbmH8bn626SmrFNcGBJASxOSqYZ1O/dD4kmvDvCOP9TMTmGCWN8MdPGBGjY19UYT7XhnObAktZHeZNd2A/5qQ9DpKGBxtFOpljOf1n7P67DXHgrRTkGpNkomjUdw/cx9Ylerq0XsYQVj8eK/yN/Z9BoqqPuhIvp43vs0OLl4305FM90YDGZsE0vxtHc4/6dLcm17t/lx3JDCa5sE2Tamfn1u3BZe2lOZ7txjQngbYgBEfy7AuROddN5S6x45j3A/Bk2TJjImejAGg4RTpx7mD5d6DX1EN3/CYEUJ9e7O2cqmJ138NBfzsSeCpw8wIfeI5jHjycnFWA017tG07bvE/50AeWJiIjIlUlzGvrS15yGNCfuCRvYvjfMqIP1WAtKsAKhcJhoYANPf7+ic7sERC3TicCpuQrdN9uMGS6kodjNNBQjPUqkt2H7xyO0GhZOTxkwYbEYtLaGgXPfhGNkGkSOnACguaWV7c8/SV13hIyBfUo/5wZYpuDM2YhvT4isfQFGFZRy/swQIXw8hsXe4+YOs2JpD9B6su+9zpCArKnTyKnejve4E+tHXpi6EHtq/ekn4+EwrSf9lP/zE5R3LYombGRHgGH9PA8AMcLHIxjjTl+ZKcdFUa/bWnB/yUb17npi+TH8B+y47urez4Spzc/GX9fR2BqDRCuHE54LKeSirikaboOhIxl29v/66AE2/fSnvNhSyHf+2/RTk5+HD8+Atjba4kDqhZUpIiIiVxaFhgtmIr/AzoadNdQdycJ5X+fQFsMYitk5n8cWec6Z0NzPPoXzOxmmNWLGZgBnt+kzLWRFAoRjQBpAjHA4gpHbexM+3NKKYQwFMrBkWpk5bxml48/aKHG4H0VZcBbkUumroi5owTnL2o99DCwWg3A4zKlI1RKiOS2LrH7Ouo0CJoubwglV1G7fTtbuLDz350C8/vRGmQZZlkLm/u08nBf1m27CkmkQOd4jAoabCESysdvOnb5ucbnIedeHd3eUwDg3Zd1vUmr3s3FtHZb/spiHxxvQVMHTz/c2+Ker2Hgvqy7imsyWDGhroeUk9HwtUnTXm7zaMJI7/+GbXN8jeHwWaoOMDDIUGERERK56Gp70ZzAmurHv20JtmhPXiNPL8hurqfCFiAHh/dVseK/p9Bj9vqSYMJ9oJdzex/pwA96GMCQi1G+poeFaJ47enihnO3Hl1rOlqp5wAiL7K9nSMApPQXcjPkrjnt0E2yEWrOWtuhiOiXmQkoOzwExdVTVNx4H2IHVvVODrcwiPGVNKhHD49JVZC5yMaqjFO9TZ55uIzpY31Ul022ZqD8UgFqLunW1EXdNwXFBD2MDpcRB8ayO7bdPwnH3ubCdO626qu+5JrNlP5cbariFjZkzmGK3h/kW6vAIH4Y+q8bcAkSaq162icl8fn26mG/foeireCmCf6jwdIqOttJ4AEjEi4SC+j+oJxWPnzhlIySJ7WIj6vUFisTCBD+to7O6ZSnpNyZnzJzORBj70tp25fPwtPLR0Mbfm9lgYP8CH3kNk5E+mny+cFRERkSuYehr60jWnofzUAhPOux9n8QwLGPk48w1CuU5OPVe3eJh/T4h1rz3Dk7+LYYxyU/IXxee/wXYXnrS1/OInJ1j0vXk4z45xaQbhrSt58vnDkO1mzr3Ffby600rx3fMJv1zOU98PwzA7RfMWUHyqIW0miz9S/u8b2R82yPtyGXcVdFZnv3UBpa+Vs2rFZiIpVuzT5zJ/GL0PoUqzUzg1wrqfPUXkv3b1TmS7cI7ZTMQxpd+vFTXlz2XhzHJe/tWTbGw3sLlmsfAOxwX/QpomTsdl2U14mruXYVE2Su6ZT+vL5ax4IgSWPDy3lWFNAbDg9ORT/coKfhlZwgM3n+c8k+ewIFhO+c+fYG3Cgn16GfOn9zUQy8DpslO+D9zOHj0RmYXcfnsD615YQU2KDfcNduzGCU6cMyTLRnFpEQ3rn+LvN2eRN82ONSVyal3f13Qe2dOZff0r/Kz8JXa4v8n1Xb0NLdtf5WebRvPIT+6loKtXoanyRd4KjuOOv5zcjwOLiIjIlW5Ic0trRyLR2TrsfsVqIh4nnkgQj8eJxWLE43Gi7e3E4jHa26Pc4Jk2wGVfJQ5V8vTzEcoe6eMNQv0Ww7fmSWrzl3WGns9dmNqVK2m+7WFKx16Cw8vnp7mOZ//pl+ywzOT+JXdz40gzNB9g1xEzeRNHk0EbgTd+xb+u28O1d/4tj319nL6nQURERNTTIBcpESPcsIX3jzmZr3Esl79sDw89upjnfr6an/1dHQU3FjF10niGp7dT89o77NxWy46DZgrKlvKdOQoMIiIi0kmhQS5K8N1f8MzvzRTfex85miEzONimc//yyXy5+nUqP/iYTdu30BKFjGGjyXN+je88OJMbczPOfxwRERG5amh4koiIiIiIJKVnwyIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk0iIiIiIhIUnp7Uh/C4WOEj7URTyQ4ETnn27dE5AuUkTGUtrYTA13GOYYMGUJmpkF29jCGpqcNdDkiIiKXjEJDL1pbjxE+dhzDMLBmZw10OSJXvb31ASY67ANdxjkSiQTh8HEOHDjIuHFjFBxEROSKpeFJvTh2vI2MDAUGEUkuJSWFYcMs2K610tzcMtDliIiIXDIKDb2IxeNkD1dgEJH+sVgyOX48MtBliIiIXDIKDb04eaJ9oEsQkUEkJSWFjo6OgS5DRETkklFoEBERERGRpBQaREREREQkKYUGERERERFJSqFBRERERESSUmgQEREREZGkFBpERERERCQphQYREREREUlKoUFERERERJJSaBARERERkaRMA12AiMjnquMg217ZQsOJrp9Tx1D0jZlclwZE/kDVhu0E413rjEnMurMQ2xAIfvgqVfVtXStMjJl+JzPz0774+kVERC5DCg0icsWJt0eIRLp+SI2fXpGI0x6JEOlelNpO9z/jsTYip3eiPd5jPxERkaucQoOIXBHaW4N8FgESR2k7o70f4bPDQYJpEI8co73nqngbnx0KkpoS52jbGYejvTXIwcMGqaRhsQ3HGHLJL0FEROSypdAgIleEo/9RxWZ/5NwV8SA73trIjt52igT4YHOglxVxgt4qNnuB1LHcPP9rTDI+33pFREQGE02EFpGrTCrGsOFYNF1BRESk3xQaROTqkmIh/+ZbmToydaArERERGTQUGkREREREJCnNaRCRq0KqubtnIZXUFBOkppJq7loUjxNPDFRlIiIilz+FBhG58qWOoXDOTCYZncEhNT0NSuaTlwCIc7D2FaoaeplELSIiIoBCg4hcDeIH2Vb+W7YBpAzHdfutDPv4Zd5r0ncxiIiI9IfmNIiIiIiISFIKDSIiIiIikpSGJ4nI1SXxGbvffBniGpokIiLSXwoNInJFGD7xK8wacwmCwJA0rPoiOBERucopNIjIFcEYORb7yIGuQkRE5MqkOQ0iIiIiIpKUQoOIiIiIiCSl0CAiIiIiIkkpNIiIiIiISFIKDSIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk09GKokT7QJYjIIJJIJBgyZMhAlyEiInLJKDT0IiUlhVBz60CXISKDRDh8nMxMY6DLEBERuWQUGnqRdU0GkUiE5s8UHESkb4lEgpaWMME/hcjOHjbQ5YiIiFwypoEu4HJksVxDRwccO97GseNtnIicHOiSRK5qGRlD2VsfGOgyzjFkyBAyMw3GjRvD0PS0gS5HRETkklFo6ENW1jVkZV0z0GWIiIiIiAw4DU8SEREREZGkFBpERERERCQphQYREREREUlKoUFERERERJJSaBARERERkaQUGkREREREJCmFBhERERERSUqhQUREREREklJoEBERERGRpBQaREREREQkKYUGERERERFJSqFBRERERESSUmgQEREREZGkFBpERERERCQphQYREREREUlKoUFERERERJJSaBARERERkaQUGkREREREJKn/D16Pyd5+6OqXAAAAAElFTkSuQmCC)

This configuration section is available only if your [Metric Type](#metric-type) is set to **Numeric Metric**. In that case, this configuration option is required to be set.

### Event property for Metric Value

In this section you can specify the event property whose value will populate the LaunchDarkly `metricValue` object.

> **Note:** Since the metric value needs to be a number (e.g. `10.0`), the Tag **will fail**, if you specify an event property whose value is not a number.

In order to specify the common event property of interest, Key Path notation can be used (e.g. `x-sp-contexts_com_acme_transaction_1.0.total` to select the `total` value from an entity (at array index 0)) as your metric value.

## Context Keys

### User Options

![User Options section with a User Value dropdown set to Common User ID (the default selection).](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxAAAAB/CAYAAAB/syTvAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABnrSURBVHic7d1/dNTVnf/xJ8kkQ6AJkkKoCAYhFhJxJVChRAtYLFGLQCs//AHoRnG1Wku/WMuyVb66FrGVo+uW2gJm1ahYpP5AsAZBgVZSQ4VaCGKbuMQA1QBBgiRkMjPsHyQQEHGQahp5Ps7JOfD5zNz7/kxyJveVe+9nWu2o2rU/Go0SjUSIRKNEIhHC4TCRSIT6UIhwJEwoVE+/7D5IkiRJOrnFNXcBkiRJkloOA4QkSZKkmBkgJEmSJMXMACFJkiQpZgYISZIkSTEzQEiSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClmBghJkiRJMTNASJIkSYqZAUKSJElSzAwQkiRJkmIWaO4C/lnV/imf/1xQytkTZ3BF78ajYdY9fgfzN2Zwxe15ZCd9TrW8W0zhsiLWl79POLET6b0HMvTC/qS3PYFGNxfyX4+u47QxP2B01ud0IZIkSWrxnIH4JxfeXMicuQtZvTVARt8css8IsLVoIb9+uJDyUKyt1FKcP43bfvocpdGGQ4lJJLdPJTmY8BlVLkmSpC+iFhEgajcv57lXy6k9+lnKX32O5ZuPfrZlq6K4cCVbAxmM+t5NXDFyOKOuvImbLs0gYdtKCtdUffqmOw8i75brye3hJJQkSZJi1wJGj1WsX7Wa1RtXsbV+AnnDMji04KaW0qX5FLxSTiCrM9nd+pP6OVe3p2w5i5esYdN71ZB8Gr3OH86ob6QfrLFqQyGLlxVTtrOWQMde5OR+h6E9kyFazuL7ZrP6S9n0TyineFs6E2+/gl5NI93uTWwoD5OcPYj+Xz50OPXcHHotK2X9xk3sOS8HXs/nnmer6DXgNPZsXM/WUAqn9f4mY77Tn7S4Up67Zw6rdwOsZs60TQy6YSrD6xYy/eG19Lj8dib2TQLCVP7lJRa/so7ynbUEvtyL/sOGk5t14BXdUzSHe56vpf+3e1G9pphNu8KknjmIMWOGkp4EUMu2oiU8+/v1bP2gnqSvNLlWSZIkfWG0gBmIVPpfPpHcHgHKXykgf2lpw0xEk/DQI5drL//8wwN71vHs44VsCvQid+wYhqbXsn7JYzz35wMVhsueI//JlWxLyWbosKFkxpVR+OR8Vu881ER483o2RdPp/7UeH61/RyVV0QCpaWmHJ73ENNLaBwjvqOT9xiVJ0UrKtiaR/e0rGfW1VCrfWMhjheWE406j/+gryOkSgLa9yB0/mv6dPnoptW8/R/6Tq9iamMmgYUPJTCxn+eP5LC4LH3pQdCvFfygnJXswg3smU7VhOYv/WHngOt5ewmPPryV8xlDGjB5OJmUUPj6f1btO8DWWJEnSP5UWMAMBJKYz9OqJ8OhjFL5SQH50DINZydMrGsLD1UPpnNgMde2upLI2QFpmDjl90qBnJ5K7llKfHAZqWftaMZXJfckbdxEZCcAZYbb+YiXrNlaRc15DG52HMnHSUDofLcqFw9RGITXxyH0KARKCwO4wB4f3cclk544i50ygT2f2VdzH4r+sZ/PFw8no2YvOrwF7UknPyiDtI33tYf1ra6lqm83Efx1N7ySgbyo1981nzWvrGdoj+2Affb+bx6ieAahN5v2357Pp75WESaN6RxXVpNL73ByyuwXI7pJC2lvVpEaP7EuSJEktWcsIEHBEiHiMx4DkM5sxPAB0zianRzHPFc5m5ls96PXVXvTul0NG+wBEy6l8Lwy7ism/s/iwp6V+sOfgvwNtk0j+uHmgQIBAHNSH6484Eaa+vuH8UZ+YSudTk2BLJVV1wCfdZCn6PtveDxM4rUfDciSgbQYZnQNsqNxKZTS7YXYkgdbBhh6DSbQJAOF66oHUrHM5e9XTrHr4PsrPzCDjzN5k988hzRs8SZIkfaG0nAABh4WI1eR8tuEh7sBLUx8Nc+hlqocoEB+AeCAujZxrb6XbhjWs3VjKptcWsnplEUOvvYnc0+sJR4DOOVwxKvuw5UkJ7U4Dtn5yDalpdIoL8/57lYRJPfTNClVRuTNMoHManT4mfIQPLm06rqtu8gPRcK2xPr99NldMPo3sP69jw9ubKH6hmJWv9WfijaPpdSK3m5UkSdI/lRawB+IIiekMvfpWbv2MZx6SOqSSGhemsnzzoWVCoa2UvxeG9p3oFIDaslUsXrKa6vRBDL88j1snjyIjspV1b26GuE6kdQjArirq26eT3u3AV3JiEu3bx5jb2veiV5cAezauZm2TvQRV61azqTZAt6xeNN2iHK5vqDS6ja0VeyA5jU5NZwA+LgzEdaJzpwDhv7976Nawtdsofy9MIC3tKEuejhRm2+uLWfz7StIG5jL6mh9w69hsEnasZV3TPRSSJElq8VrWDESjxKRPXJVzwrr0J6fHGp57bT6/3tefXh1g11vFFFcl0es75x7YsxCspaxoORu2h/lm3zTYuoH3owE6dUgFkske1JdV/1PMcw8/RmW/biRVbWL1mmqyr5/M8NNjKSKVnEsGs+7h5Tz3qzm8+y/ptK4pZ8O6Uuq/MpTcc5vMa0T3sPbFJ0nemUGgYg3LtwVI++bZpMcBJNCmTQLsLWX10lWEzx1Er8P6Sebsr/fllceKefrRAIOzUqnasJJ1tanknHc2ScAejiVA6/A21ryyjvLaanLOCFD1l3Jq4zqR2qFl/ohJkiTp6BzdfZy4NHLG58FLhazesJLltZCUlkHO6IsY/rWGgXuXXCaMh8XLilmyoJb6tp3o9c2JDB944HxSz1Fcf2Uyi18pZnXhJhJO6cbZF49haLdAzEuDAt1yuf7aZApfWc364pWEE1M57dxRXDEsh/SmMzBxSXQ7ow2lfyhka00Spw0YxZgL0xtbodf5Q+n17nI2Fa2jU68jAwQkZY0i7/LWLH5lHS8tCZOUls6gK0dx0ZmxRbXU865gQmgxhWsKefr1epI6ZDBo7HCGdo7tOiVJktQytNpRtWt/NBolGokQiUaJRCKEw2EikQj1oRDhSJhQqJ5+2X2au1Z9jD2v53PPs1vpf+3tjDqzuauRJEnSF1nL2wMhSZIkqdkYICRJkiTFzCVMkiRJkmLmDIQkSZKkmBkgJEmSJMXMACFJkiQpZgYISZIkSTEzQEiSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClmBghJkiRJMTNASJIkSYqZAUKSJElSzAwQkiRJkmJmgJAkSZIUMwOEJEmSpJgZIHT86uubuwJJkiQ1k0BzF6BPo54539/D9mvb8x99Wn0O/dWw7fVlLHp1DW9u3kFNPZCQROduvfnaBZdw8YCv0OZzqEKSJEnNzxmIRv/7IWOu2cPrkSbHttRw1bhqloU+g/5CIe6btJM7/rj/8OPlexkz7gOe2fUZ9Plp1LzD7x64k3+ft4Jt7fowYvz1fP+m6/n++Es4p917LJt3J1MfWEZ5TXMXKkmSpM+DAaK5JCbwrf5xvF4Uoq7J4Xf+GGLLWa35Vvtmq+yQ+vf43YO/4Kl3T+XyqXdz+7c7UL60gP+ePYeHl77LqSN+xKypI+j27m+Z+eAytkU+uUlJkiS1bAaImO3nrUXVjJmwk37jdnLVrFreaZyZiERYNvcDRly5k4F5H3DfqoaRdCTET/N28b27P2DIuF08sqVpe604Z1ACwfUh/nxwhiPMquIoA76RSDLA3nrmz9pF7pU7GXjNLu5YHj5KXVGW3LWTyUujB9t4ZEqTmY2Pqy0G218t4Lfvnsplk2/i4tPf47cPPcWbbc4j7/rxDG6zgfxfLGJbt4v5/uQRdH53EQ8v3xFz25IkSWqZDBCx+nstdz+9nyt+lsobc5MZvK2G+5ZGgP28s3APP38nkXvnprL01gTeyv+QZ3Y2PC8aZfepbVj46Clc0+WINjOCXJhUz7KNDYP9v4f4fWUCw/vHAft588kPmU8STxR8maWTE3jr4b0sqT6eoj+htmOJVLBq1Tuccv5lXHx6AtTsJuGMwUzMu4zBA87j8jHn0XHX2/xtJyScfiFXnN+O0lUrKXcWQpIk6QvNABGreGjNfrZsibCnbSLX33UK914YB5Ewy1ZFGTQ6icy2rUjOSuLb6WFeL2kIBXFxDL4wkQ6tj7LZOT7A8K+34vU/Hrir0dY/hthydiLfaAvQiu7Dv8RDN7SmQzwkn51AZnyUrcezN+KTajuW6ncofT+FzHO6kwDQLpvLrr+cgWkANZT+6W0++NLppLcDSCDjnJ60e/8d/vbhcdQnSZKkFse7MMUqLYl7b9nPg0/v4dIHIevrrbn12iSSibKlOsqSn1WxuPGxUcj+evRYrTVoRfdBiSTfE+LNUDwbi6MMGBkk2HA2OT7K/Nkf8HL5fmA/2/e14srjKvpYtcUf+6k1NeyNb0NG8pEn6il9+gFmroSLb/kuWQeLTaFt/LvUfAi0O64iJUmS1IIYIBq1jiMxFGZPPYfG1nuj7EmE5HigOsL2Lknc9bO2sLeeR+6p5vZnAjwxLo4ObeO5+iencEvPI2YZIjGs50kP8q22e3h5TRzvvB/g6r6NbUR4ZvaH/PnsFB6/LYFgJMRPr917lAYOPD501K6OUdsn+VIb2kZ288HuejgwB3HAh8UsWv4eZ467g8t6Nbl56+5qPoi0oc2Xjq8bSZIktSwuYWqUlsDgL9fzP0/UsXXffup21TP/NyHq+gTJioe6v9Uw+c4PWVa5HxLi6Ni21YFBe3yAwefCkidreat6P+wLs+SRD1lWGWvHAb51fitefngfW/q1pk9i4/H9VH8IdUBob4S3lu/j9b1A9KPLj7qktaLszyG27tvPjjf28XLjZu0Tqa1ddzJOreXNte9w2MfGJfRkxI03MSGnQ5OD9Wxcu56aU7uT5eyDJEnSF5oBolF8AtdPa0v2/9Zw1YSd5Nywh2VtW/PADUGSgWC/ttw1aD+/nlJFv6s+4JFIa37y3QSgFedM+BK3fDnE5ElVDJy0hyWRBPqkxd71aecl0mUPDBqUcHD5EgS4bEJrkpdVMyxvN/dtimPAGVC9+8hnt+Kc77Zh0Pt7GXFlFXkLo3Ts0OTcp66tK4MG96Sm6LcsKmsSIXas5uGHnuJPTW64VF/2O54qqiVrcA6dY79sSZIktUCtdlTt2h+NRolGIkSiUSKRCOFwmEgkQn0oRDgSJhSqp192n+auVZ+3yA6WPTCDgndPZcT1/8ZlZ6VA3Q5KS6tpl9GdjkHYXbKIX835HZtPv5y7Jg+m4ydsrZAkSVLLZoDQsdVVsCx/Dk/9aTenZPVn4Nln0LldIqHdf6d0/RqKNu7mlK9dxv/LG0zn4Cc3J0mSpJbNAKEY1LP9zZW8tLKYN8ve44OaemjTgc49ejJw8EVceE6HptusJUmS9AVmgJAkSZIUMzdRS5IkSYqZAUKSJElSzAwQkiRJkmJmgJAkSZIUs0BzF/DPZGNZmF/9ppbX1oWoC330E58lCSCY2IrzshO5YVwSWT18G5UknVz8zddgY1mYCVN3GxwkfaK60H5eeb2O19aFKJjZzhAhSTqp+Fuvwa9+U0tdaD/fGdqaH17dhtR2ru6SdHRVu6Pc/2gNzy7fx69+U8uD05KbuyRJkj43jpIbvLYuBGB4kPSJUtvF8cOr2wCH3jskSTpZOFJu0Lh0yfAgKRaN7xUue5QknWwcLUuSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClmBghJkiRJMTNASJIkSYqZAeIfoHpdAdPzhjGgTyaZA4YxfloBa6ubu6rPyIqp9Ok9mcLI4Ye3P5VH5gUzWBs5+tP+YSKFTO7dh8nLD/y37sXJZHbvTvfu3eme2YcBF41n6pwiKj7rOiRJkk5SBogTVLd+Nnn/+gvKut/MA0++wAsP3ExW6Szybi6g1EHs5yMll5kvLWXp4id44JaB7HvmRsb/YBEVzV2XJEnSF1CguQto2bbz/INzqRhyLy//JJcUADKY1rmStd+ex4LiCUwb2Mwlngzi25HWI4OMeMjocRYD/6UjeSN/yuzlQ5g5NKW5q5MkSfpCcQbiRFQX8WpRkAsuHcJhw9T0XCZdm0saDeuYakpYcPt4hvXLJLPfMMbfvoCSmsY2FpHX+1Kmz5lF3kUDyOzdh2GTZlNUXkLBbaMZ3CeTzAGXMvmJEuoAqGDe2EzG3z2P6ROH0ad3JgOGT6ZgfQVFD93IpTmZZPYZzOhpiw5bxlNXXsiMScMY0DuTPoNGMzl/bWN1rJ05jMwb57Ho7oYaB1zK5Kca+/uUL836AqaOHUyfzEz6XDCeqU3bqyk99HoMGEbez1ccrHXtzGFkjp3K9ImD6dP7RhbVfFwPx9BlJOOG1PHSi0UndA2SJEn6KAPEiXivgi2k0bVr8IgTXcn94TSuG5gCbGfRbXnMeDuLKQVLWfrIFLI2ziDvtkVsb3x4pISX/gDjfv4EL8ydQlbpLG787g95teskfrlgIfnXdKTonnsoKD/Uw+qlb9A1734WLniIiWlFzJhwKTNK+jFl3gss/FkuvHgHMxY1RITqFcy45scUfeVmHlqylIV3XcCOX+fxvUcPLfKpW1HAc+0mcP/8hTw0vh0r7p7F89v5dOqKmHXLLEr73snCl5eSf1MX1t79PWb9qQ7YTuHt1zL7vSFMm7+UFx6YQMqLk/n3prWsX8vuoTN54pm7GNLm0xQQpEdGBnWbS13GJACeXBzhkhv2HfPrycWuOZQkKRYuYToR4X1AkGD8MR7z1wXMXZHGpGemkftVgK5M+89SisY+xvNlI7iuIxDflZG3TCH3bIAMJl1SQOEfRjLte7lkxAPpE7lg3mRKS4H0A82edcUUrhuSAUDGxAt4rHgLY6dfx5COQNbNjH26gHlvlwJ92f7iYyxgLPk/GUHfIJB+E/ffUsSgRwpYO34aAMFzr+Pe7+fSEaDLWAb+ehYlpXDgwHGq207F9nacdf5AMroEYfQ07g++xJYUoOx55i7vwU0vN9TKBKZcU8iwJYVU5F13oJaB1/EfVw/8VF03CrZpDXv3URcBjvX90UnhyuEHfggeX1x/1PPjhyccfIwkSTo2A8SJCLQG6g4MUj9GXekGSlP60rdHk4M9+9K37Vw2lNU1DNCDhw1yg62D0C6Fdo3HAkGCwTp2N+knGN9k1iMYpDXJpB1cRxWkdRDqwgcW8JSWlMBZ4w6EhwYds7Ppek8JpVWQ0dBGsEl7wcC+Y17XMaUMYdLVjzH55mGUDMnlgvMvYMQlY8ltA3VLN1BSU8T0CzKZfuhVoq5jFtsb+4uHI+d0jlddzT5o2/rY4U4nlY8LEYYHSZKOjwHiRHylK12opKKiDr7adMhbwYo5C6g8dxIjCR71Vf5c1+YHWn+0hghA6FO0BUHqIMzhf9mPNG0rhYE/WsiqsUUULl/NKwumcunsLKY98hAjw0D7XO58cgoDm9YUbEdaPJQcf0VHUUfZ26UEu06i6z+kPX1RHBkiDA+SJB0/90CciJSBXNC/jldfXMFhH/uwpZB5Dy6iZF8KwYwzyagqoWRzk/Nvr6Vkb1fO7HGif2ePTUbPHlDyBiVNUsv2v6yjomMWGanH2dipGXSljA1/bXqwjtJNFdC1K13jgbJCZv93IRVdBjIibwoPzH+YSakrKHixlGC3rmTsLWN7XVe6pjd+pZHWPuWEZx0OKl9AwYogQ4YN/Me1qS+MK4fHM354guFBkqRPyQBxQjoy8vsTSFt+B9+7v5C1fy2ltGgRM348lw19r2NCf+CrY5nwjQrm3jGLFX+toGJjITP+/1wqvzGJsT0+sYN/TJWXTGRkeAHTZy6iZEsFpStm8+PZG+g3YSx9j3f81GMkY7+xm4KfTGXBH0ooLSthRf6Pmf4sjLx6ZMOKrEreeOQOZvxyxYH+igp5Y0uQLl9Jg6yxTOhfwdzbprLgT6VUlK1lwW2jufTnJ3DHpMhuKjdXUFFWQtELs7nxmhn8beAUpuR6C1cd3ZXD4w0PkiR9Si5hOkHB7CnkP9SOn94/g7w5ldSldKXfsJvJ/9GEAxug6cjYe/Opmzmd6WPnURnflX6XTCP/RyNOaJPwcUkZwp3z7mLGnb/gqosqoONZDLnmIe7Ky/gUjXVk7Kx8+Pks5k0ZTeku6PjVgYz82cNMafzMhS4TuPe/qvnp/dO56peV1LXPYMj4B7hrRAqQwtgH8qm7ewaz/+1SpkfSOGvoBGbedAKzBdWFTP1WIQSCpKT3Y8jYh1iYN+TAbIgkSZL+oVrtqNq1PxqNEo1EiESjRCIRwuEwkUiE+lCIcCRMKFRPv+w+zV3rZ6r3yB0AbHi+QzNXIqml8H1DknQycgmTJEmSpJgZICRJkiTFzAAhSZIkKWYGCEmSJEkxM0BIkiRJipkBQpIkSVLMDBCSJEmSYmaAkCRJkhQzA4QkSZKkmBkgJEmSJMXMANEgmNgKgKrd0WauRFJL0Phe0fjeIUnSycIA0eC87EQA7n+0xhAh6Ziqdke5/9Ea4NB7hyRJJ4tAcxfwz+KGcUm8ti7Es8v38ezyfc1djqQWIJjYihvGJTV3GZIkfa6cgWiQ1SNAwcx2fHNA0CUJko4pmNiKbw4IUjCzHVk9/DuMJOnk0mpH1a790WiUaCRCJBolEokQDoeJRCLUh0KEI2FCoXr6Zfdp7lolSZIkNTNnICRJkiTFzAAhSZIkKWYGCEmSJEkxM0BIkiRJipkBQpIkSVLMDBCSJEmSYmaAkCRJkhQzA4QkSZKkmBkgJEmSJMXMACFJkiQpZgYISZIkSTEzQEiSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClm/we1kgrkjy0gngAAAABJRU5ErkJggg==)

This section allows you to configure how to populate the value of the user property that uniquely identifies the context that the LaunchDarkly metric is about.

#### User Value

With this drop-down option you can select how to derive the value for the `user` in your `contextKeys`. The available options are:

1. **Common User ID** (default): Using this option the Tag will populate the `user` from the `user_id` property of the common event.
2. **Custom**: This option allows you to specify an alternative property of the event to be used. Selecting this option reveals the following text-box to specify which event property to use.
3. **Do not populate**: Selecting this option will not populate `user` as a context key.

#### Event property for user context key

![User Options section with User Value dropdown set to Custom, revealing an additional empty text input field labeled Event property for user context key.](/assets/images/05-user-value-custom-ffce8f0a70afdfd5304fbbf846857140.png)

This option is revealed if you have previously selected **Custom** as the option for the [User Value](#user-value). In this text box you can specify the Property Key from the GTM Event to use. You can use Key Path notation here if you want to denote a nested key (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1.0.userId` to use the `userId` from the Snowplow client session entity (in array index 0)).

### Other Context Keys

![Other Context Keys section showing an empty Context Keys to Add table with Key and Value columns and an Add Row button.](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAw0AAACoCAYAAACi2AIgAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d15XJV13v/xN5zD4bBv4gIqCBhu4EIuWZZmi+aSs2lp3U273TX31Ey3U/O457ZZ77znnt/M1My0aDXlWOqUMZmKImLiCqJOouCGZCoqyr4fruv8/mA4RcCFlkbU6/l48ACu73W+388BH57rfb7f74VXRVW1W5LcbrdatHxtmqbns9vtlmEYMgxDpmHIMA25XE1yNTbKMAzVNzRodMooAQAAAPh68e7qAgAAAAB8tREaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEaAAAAAFiyd3UB+OL27t2rzZs36/z58woJCdHYsWN1ww03WD5m6dKlOnDggJ599tkrWlthYaHWr1+vU6dOyd/fX4mJiZo6dar8/f2v6LhXSlFRkf7yl7/o1ltv1eTJkyVJlZWV+v3vfy/DMPT4448rPDy8i6sEAAC4vJhp6Oa2b9+ut956Sy6XSykpKXI6nVqzZo1SU1M95xQVFWnBggXKysr6UmsrLCzU4sWLVVZWprFjxyo+Pl45OTl69dVXZZrmZRkjIyNDCxYs0OnTp7ukP9M0tWzZMtXU1GjOnDkEBgAA8LXETMNlUlRUpCNHjujmm29utz09PV0DBw5UbGzsZRuzvr5e69atU69evfSDH/xADodDpmnqr3/9q3bs2KFx48apd+/el228jpimKW/vtvlz9erVstvteuyxxxQcHCxJ6tWrl9auXav8/HwNHTr0itd2paWlpen48eO6/vrrvxbPBwAAoD22p3/602c6anS73W0+f/rDNE2ZhiG3260mw1B0VJ8vpeivonfeeUfZ2dkyDEMJCQmt2tatW6eMjAxVVVVp5MiRl23MgwcPau/evbrllls8YcTLy0tBQUHKzc1VYGCgTp8+rWXLlkmSDh8+rOPHjyslJUUffvihSkpKFBAQoKVLl2rTpk1yuVyKj4/39J+Zmak333xTGRkZKioq0oABA+R0OnXu3Dn9/Oc/1/nz57V+/Xrl5+dr1KhRrWorLy/X2rVrNWLECKWkpHiOh4SEqKysTBEREerTp4/cbrc2btyo5cuXKy0tTceOHVPfvn0VGBgoqXkZ1T/+8Q9J0htvvKHMzEzV19crISFBr7/+unbs2CFJ2rlzp7y9vRUXF2dZ+/bt2/X8888rJCRE0dHR2rt3r37/+9/Lbrdry5YtHfb36eeVk5OjhIQENTY2KjU1VTExMbrzzjtbBaeDBw9q6dKlWrNmjQ4cOKCePXsqLCxMGzZs0EsvvaTY2FhFRETINE0tXLhQH374ocaNG6fy8nKtWLFCq1atUlZWlkpLS5WQkCCbzfaF/70AAAB8XixPukzuvvtuxcbGatOmTUpLS/McX7dunTIzMxUbG6u77rrrso5ZUlIiSW1mE3r27ClJOn/+vIYMGaKpU6dKkkaPHq1bb73Vc55pmsrOztaYMWMUFBSk9PR0nTx5UpK0ceNGrVu3ToMHD9bEiRN14sQJvfHGG63G2bt3ryIiIjR48OA2tZWWlkqSIiIiWh0PCwvTPffc4wkZ69evV3p6ugYMGKDJkyeruLhYS5YsUWNjo+cx1dXV2rt3r8aPH6/g4GBt2rRJH3/8sSZNmuQJYTNmzFBycnKntY8bN069evXSxo0bVV9fr7S0NIWHh2vChAkd9teeiooKLV++XE6nU3fddVeri/rCwkK9/vrrCgoK0i233CLTNPXKK6+osrLS02d+fr4k6aOPPlJ9fb2SkpIkScuXL1dRUZGmT5+usWPHaseOHa3+PQEAAHQFliddJg6HQw888ICWLFmiTZs2SWq+KN+8ebNiY2P1wAMPyOFwXNYx6+vrJUl+fn6tjrd839DQoIiICA0YMEBSc7iIiYnxnOfl5aWHH35YTqdT/fr10yuvvKKTJ08qKipKWVlZGjJkiKZPny5JstvtWr16tc6cOeN5R/3qq6/W7Nmz262toaFBkuTr69th/U1NTdq2bZvi4+M1d+5cSVJ0dLReffVV7d69W+PHj29Vp5+fn/r3768lS5bo1KlTGjdunI4cOSJJio+PV2RkpEzTtKy9d+/emjFjhpYsWaIXX3xRZWVluvvuu2W329W/f/82/XVk165dcrvdstlsqqurU0hIiKftgw8+UEBAgObNmye73a64uDg999xz2r9/v6699lr16tVLBQUFmjlzpg4dOiRJntBw+vRp9evXT2PGjJEkBQcHW/4MAQAAvgyEhsuoveBwpQKD9MkFeV1dXavjLd87nU7Lx3t5eXnO+XTQKCsrU11dnQ4ePKj/+q//avWY0tJS9ejRQ5JaXSh/VsvzbQkP7SktLVVDQ4Mn1EjyLI86depUqzpb6mupt6N+O6u9d+/euuqqqzRo0CAVFBQoLi7Oc8F+Kdxut1JSUpSbm6u3335bjz76qLy8vCQ1X/hXV1frmWeeaVObJCUnJys9PV0lJSU6dOiQevfu7Qko1157rTIyMrRo0SINHjxYw4cPbxX0AAAAugKh4TL7dHCQdMUCgyTPxfvZs2dbXXi3LFtqaf+8kpKSNGHChFbHevbsqerq6k4fGxYWJumTC+UWFRUVSk9PV2JioqKjoyXJc7H9aV/07kod1d6iJVjV1dXJ7Xa3W4OVlJQUzZkzR97e3srJydG2bdt03XXXedojIiI0Z86cVo9p2QzeEhp2796t06dP66abbvKcc+uttyoxMVH79u3TgQMHtHXrVk2bNq3TW+gCAABcSexpuAJagsOVDAySlJiYKIfDoe3bt7faA7B161Z5eXlp2LBhrc5v2dDembCwMDkcDp07d04xMTGKjY1VbGys7Hb7Rf99hfDwcPXq1Ut5eXmqqanxHN+3b5+ys7MlSaGhofL19dXx48c97S1f9+lzaZvqW57bxdS+b98+ffTRRxo0aJCKi4s99bTXX0daAtn06dMVGBio9evXq7y8XFLzHaLKy8sVGhrqGd9ms3lux9qrVy/16tVLWVlZcrvdnpmOiooKpaamyjAMzZo1Sz/5yU8UGRmp3bt3X9LPAgAA4HIjNFwhDofjigYGqXlJ0ZQpU3TmzBk9//zzWr16tV544QUdOHBAY8eO9WyQbnmHe9++fRd1Aert7a2JEyfq7NmzWrx4sbZs2aLFixfr5ZdfbhVOOjN9+nQ1NDTo+eef15o1a7RixQqlpaWpX79+GjJkiLy9vXX99dfr6NGjevPNN7Vp0yatWLFC/v7+re64ZKXluWVmZuro0aOd1u5yubR27VpFRkbqnnvuUXR0tNLS0jz7Qz7bX2f8/Pw0c+ZMNTQ0aNWqVZKkyZMnyzAMvfzyy9q8ebPeeecd/elPf2oVjpKTk9XU1KTIyEjP7ykgIEB5eXlauXKlcnJytHPnTpWXlysqKuqif+YAAABXAqGhm7vuuus0Z84ceXl5aceOHaqoqNCUKVP0rW99y3NOeHi4xo8fr3PnzikvL++i+r3xxht18803q6SkROvWrVNtba3uuuuuSwpCiYmJuvfeexUQEKDt27fr2LFjGj16tO6//37P3YYmT56sSZMm6ejRo8rIyFDPnj01f/58BQQEXNQYycnJiouLU35+vufOT1a1b968WeXl5ZoyZYpsNptuu+021dTUKD09vcP+OjNixAjPHom9e/dqwIABuvvuu2Wz2bR+/XodPnxY06ZNa7WErGV24dP7Kex2ux588EFFREQoNTVVGzZsUFJSkm6//faLqgMAAOBK8aqoqnZLrZdjtHzdsq7cNE253W4ZhiHDMGQahgzTkMvVJFdjowzDUH1Dg0anjGpnCACf9c9//lPLli3TD3/4Q8/eDgAAgK8qNkIDXyKXy6WMjAzl5OQoOjqawAAAALoFlicBX6KmpiZt3bpVgYGBuuOOO7q6HAAAgIvC8iQAAAAAlphpAAAAAGCJ0AAAAADAEqEBAAAAgCVCAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGCJ0AAAAADAEqEBAAAAgCVCAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGDJ3tUFALBWWVmpQ4ePqqKiQk4/Pw2I7a/oqKiuLgsAAHyDEBqAr6jKqiq9teJt7dyVI5vNpuDgINXW1qm+vl4DYmN097w7FDcgtqvLBAAA3wCEBuAyWbp0qQYOHKhx48ZdUlt7zpWUaNFv/yCbzaZ/n/+ARg5Plt1ul9vtVuHxIr37j9X6zaLf6ZGH7lPKqJGX+6kAAAC0wp6GbsYwDL3//vv6xS9+oYULF+qNN95QRUXFF+738OHDOnz48BXr48iRI1q0aJHne8Mw9NJLL2nVqlVfaMzOGIah9PR0GYbxuftYv369fve7313Gqqw1NTXpD8+/oODgIC382VMaOniQFr/yVz3ygyf0zC//R5L048d/oIk3TNBLS17TyVOnv7TaAADANxOhoZt59913dfLkST366KN6+umnFRoaqqVLl37hfq90aPisd955R97e3po1a9YXGrMzlyM07N27V5WVlTp16tRlrKxjmR9k6cKFC/qPR+crwN9ff3tzhQ4fOabZ3/mWQkKC9Yfn/6L6+gbNu+N76te3r1a+/e6XUhcAAPjmYnlSN1JbW6vc3Fz953/+p8LDwyVJ06ZN069+9StduHBBERER+uijj/Tuu+/qwoULiomJ0Xe/+12FhoZKkp555hndeOON2rZtmxobGzVz5kyNHDlSy5cv1/79+yU1b7qdO3euTNPUmjVrlJOTo8DAQM2YMUODBw9Wdna2tm7dqscff1yGYei3v/2tZs6cqby8vDZ9dGTTpk06ceKEHnvsMXl7N+fWjsZbsmSJEhISNHHiREnSn//8Z40bN04pKSnatWuX0tPT5XK5lJSUpG9/+9ue/iSpuLhYixcvliQ9++yzuv/++xUdHa3s7Gxt3LjR87gZM2bIx8en3VqLiork7e2ta665Rnv27FF0dLSnzeVy6e9//7sKCgoUHR2tpqami2rrzM5dORp/zTiFhTX/3k6dLtadc76rMaNTNPrqFD32+JP6+ORJXTUwQdOm3qo/vfCyampqFBAQcNFjAAAAXApmGrqRM2fOKCAgwBMYJMlms2nhwoWKiIhQXV2dXnvtNU2aNEkLFy5UTEyM3njjjVZ9nDp1Sj/60Y90++2367333pMk3XHHHbrmmmt0zTXXeC72t2zZolOnTunpp5/WnDlztGLFCtXV1WnMmDHy8fHR7t27tWXLFvXs2VPDhg1rt4/2fPjhh8rKytJ9990np9PpOd7ReElJSTpw4ICk5tB08uRJDRo0SBUVFVqzZo0effRRPfXUUzp9+rR2797daqw+ffroqaeekiQ99dRTio6O1vHjx5WWlqZ7771XP/nJT1ReXq7169d3WO+ePXs0bNgwJScna+/evTJN09OWmZmpyspKLViwQFOmTFFJSclFtXXm5KlTGpgQ5/n+5//9U40ZnSJJytq6XUGBgeob3Xz3pISEOJmmqeLisxfdPwAAwKUiNHQjDQ0N8vPz67C9oKBAkZGRGj58uOx2uyZPnqyysjKdPfvJBeXkyZPl6+urIUOGqKamRo2Nje32lZubq0mTJsnPz08xMTGKjo5WYWGhJGnWrFnasGGDsrKydPvtt190/dXV1Vq1apXcbrdsNttFjTd06FCdPHlS1dXVOnz4sGJiYhQQECCbzSa3261z587J19dXDz30kJKTkzutYc+ePRozZoz69Okjp9OpqVOnKjc3t91zDcPQhx9+qKSkJEVFRcnhcOjIkSOe9ry8PE2aNEmBgYGKiYlRfHz8RbV1psnVJIePo83x99emadU/Vus/Hpsvf39/SZLD0Xxeo6v93yMAAMDlwPKkbiQgIED19fUdtldXVyssLMzzvbe3t0JCQlReXq5evXq1OrdlGU9Ha/0rKyv1t7/9TV5eXp7zWi7K+/Xrp7CwMAUEBCgyMvKi63e5XLrrrruUn5+v5cuX66GHHvL039F4gYGB6t+/v/Lz83Xs2DENGTJEkhQYGKh77rlHGzdu1FtvvaXhw4dr+vTpndZQVVXVaolRWFiYampq5HK52ixRys/Pl8PhUL9+/SRJSUlJys3NVWJioqTmn3fL0q/PsmrrTGhYqM6cbTtzsGbdBt0+4zYNTPgkgJw903xeeHhYm/MBAAAuF0JDN9KjRw/V1NSorKzMEw4Mw9CiRYs0f/58hYaGevYVSM37BCoqKhQYGHjJYwUFBel73/ueYmJi2rQdO3ZMVVVVKisrU3Fxsfr06XNRfYaFhSkxMVHx8fF67rnnlJmZqRtvvLHT8YYNG6YDBw7oxIkTuuWWWyQ1X5SHh4dr/vz5qq2t1dKlS7Vjxw5df/31ljWEhoa2uttUWVmZnE5nu3sa9uzZo/Lyci1YsMBzzMfHRw0NDfL19VVQUJBqa2vbHceqrTPJw4Zqx85sTZt6qydEud1uff/uuUq8amCrc7ft2KUePSLU+zOhEAAA4HJieVI34u/vr6uvvlpvv/22ysvLVVdXp3Xr1ik4OFjh4eG66qqrdP78eeXl5ckwDH3wwQcKDAxs9c56RxwOh8rKyjwbdocPH64NGzaotrZWtbW1Wr16tSoqKmQYhlJTUzV16lTddNNNrW6Z+tk+OmK32zV37lxlZmbqxIkTluNJze/w5+fnKzAw0LOf4/Tp03rxxRdVUVEhX19fOZ3Odse12+3y9vbWhQsXZJqmRo0apezsbJ09e1aNjY3asGGDRo5s+3cO6urqVFBQoB/96Ef63//9X89HWFiYJ5gNHTpUW7dulWmaKi8v9zyXzto6M+WWm3Tm7DmtXpPmOWaabr3w8isq+uiTfo4eLdSmzVs0/bYpF903AADA52F7+qc/faajRrfb3ebzpz9M05RpGHK73WoyDEVHXdw7zvj8EhISdOLECaWmpmrbtm3y9/fX7Nmz5XQ6ZbfbFR8fr7Vr12rt2rVqbGzUvHnzPOvfN2/erNGjRysgIECmaSojI0OTJk2Sj4+PAgMDtWnTJhUVFWnEiBGKiYlRcXGxVq1apa1bt6pv375KSkrSli1bVFpaqpkzZyoqKkpbt26Vj4+PoqOj2/TxaaWlpcrPz9d1110nqXl5kdPp1Pvvv6/Ro0crLi6u3fEkyel0Kj8/X4MHD1ZCQoIkKSIiQm63WytWrNDmzZsVGRmpqVOnttkr4eXlpaqqKqWmpmrgwIGKiYlRYGCgVq1apYyMDEVFRWnmzJltHrdnzx5VV1d7ZkJaGIah/fv3KyUlxbNs6r333tORI0fk5+enPn36qG/fvpZtnQkMDFBISLCWr3xHTUaTrhqYILvdplkzp6t37+YZhdw9e/XnFxcrOWmovvedWZ4ZCQAAgCvBq6Kq2i19Egw+/XXLnWJM05Tb7ZZhGDIMQ6ZhyDANuVxNcjU2yjAM1Tc0aHTKqC54CvgmeOmllzRt2rSLuuj+uti6bYeWvrlcfn5+Gjk8WeHh4aqtq9XBgwU68fFJTbz+Ot01d06bwAMAAHC5sacBX2mGYaiwsFCVlZXfqMAgSddde42Shg3V5i1Zyi84rEOHj8jp56eE+Dh9/9/maUBs2/0fAAAAVwIzDfhKy8zMVFZWlubNm3dJty0FAADA5UNoAAAAAGCJuycBAAAAsERoAAAAAGCJ0AAAAADAEqEBAAAAgCVCAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGCJ0AAAwNdIamqq0tPTu7oMAF8zhIZuxDRNLViwQBUVFZ5jaWlp+uMf/yiXy9WFlQEALqdly5bp/fffb3P8l7/8pQ4ePNgFFQH4piM0dGO5ubnKzc3VvffeKx8fn64uBwBwmYwcOVJ5eXmtjhUVFampqUmJiYldVBWAbzJ7VxeAz6ewsFDvvfeeHn74YQUHB3uO7969W+vXr5dpmho/frwmT56szMxMHT9+XPfdd5+k5tmJyspKzZ49u6vKBwBYSExMVH19vYqLi9WnTx9JUl5enpKTk2Wz2SRJWVlZ2rx5s0zT1OjRo3Xbbbe16eeDDz7QmTNnNGfOHEnyzF5Mnz5dUvuvGQDQHmYauqGSkhItXbpUs2fPVlRUlOf4iRMntHbtWj344IP64Q9/qNzcXB09elRJSUk6evSoGhsbJUkFBQUaOnRoV5UPAOiEzWZTUlJSq9mGvLw8jRgxQpJ04MAB7dy5Uz/+8Y/1xBNPKCcnR4WFhZc0RkevGQDQHkJDN/Tmm2+qsbHR825Tiz179ujqq69Wz549FRwcrKuvvlr5+fnq0aOHIiIidOjQIVVWVqqkpERXXXVVF1UPALgYI0aM8ISG06dPyzAMxcXFSZJiYmL04IMPyt/fX8HBwerfv7/OnTt3Sf139JoBAO1heVI3NHbsWEVFRWnlypV64oknFBQUJEmqqKjQ4cOHtXPnTkmS2+32zCgkJSXpwIEDqq+v18CBA9kDAQBfcXFxcaqtrVVpaan279+vESNGyMvLS5Lk7++v9PR0FRQUyDRNlZWVafDgwZfUv9VrBgB8FqGhGxo3bpxCQkJ06NAhrVixQg888IAkKTg4WDfddJMmTZrU5jHDhg3Tyy+/rIaGBl4UAKAb8PLy0vDhw7V//37l5eXpzjvv9LRt27ZNxcXFeuSRR+RwOPT666+324fNZpPb7W63zeo1AwA+i+VJ3djMmTNVVlamLVu2SJKSk5O1fft2nT17Vi6XS1u3btXhw4clSVFRUXI6ncrPz7/kd6MAAF1jxIgR2r59u9xud6s9bFVVVZIkwzB07NgxFRUVyTTNNo+PiIhQUVGRqqurVVxcrP3793varF4zAOCzmGnoxhwOh+bOnasXX3xRCQkJio+P10033aTXXntNNTU1SkhI0KhRozznDxs2TCdOnFBgYGAXVg0AuFh9+/aV3W73bIBuMWHCBC1btky//vWvFRsbq8TERDU0NLR5/KBBg7Rv3z4tWrRIERERrYJHZ68ZAPBpXhVV1W5JraYvW75uedfCNE253W4ZhiHDMGQahgzTkMvVJFdjowzDUH1Dg0an8J/NV1l6erocDoduuOGGri4FAAAA3QjLk74BTNPUhQsXtHv3biUnJ3d1OQAAAOhmWJ70DXDixAktWbJEt9xyi8LCwrq6HAAAAHQzLE8CAAAAYInlSQAAAAAsERoAAAAAWCI0AAAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS4QGAAAAAJYIDQAAAAAsERoAAAAAWCI0AAAAALBk7+oCcGkaGhp1obRcNbV1XV0KLAT4+ykiPFS+vo6uLgUAAOALIzR0Iw0NjTpxsliRPcIV1adnV5cDC+UVVTpxslj9+/YhOAAAgG6P5UndyIXSckX2CFdoSFBXl4JOhIYEKbJHuC6Ulnd1KQAAAF8YoaEbqamtIzB0I6EhQSwjAwAAXwuEBgAAAACWCA0AAAAALBEaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEa0KF9H7sU/eRZjfx5iZrMzs9vMqXoJ8/qjxtrOjwn60ijop88q+zjrnbbx//PeUU/eVbRT55V3yfP6prfnNerW2s/71MAAADAZWDv6gLw1bUyp16+di+dqzKVWdCgm4f4finjDu/no++P91OTKW082KCfpVbJ18dL88b6fSnjAwAAoDVmGtCuxia3UvfW63ujnYqLtGlFdt2XNnZ0mLdmj/bT3LF+WvL9UMVH2vTWri9vfAAAALTGTAPalZbXoIo6U7NGONUzyFvPZdToQrWpiMBPcmZOkUu/fr9K+081qU+ItxZMCZT9MzH0rew6vZBZo1PlpoZF2zVrpPOS6vD2ksIDvNVotK7t+YwaFZxpUrDTS7clO/XU1EAFOb30/zZU63cbarTjpz3UP9ymynq3kheeU1ykXZuejJAkrf5nveYvrdDf54dpfILjc/+MAAAAvimYaUC7VuTUKTrUpnFxDn1nlJ+aDGnVnnpP+6EzTZrzYpnKat36zbeDdP8Ef/0stUrGp/Y+vJVdpydXViomwq7fzQ7W+HiHfrG6utOxy2rcKq0xda7K1Bvb65T7kUtThjYvjXpvX73u/2u5Qvy89H+zg3XfBH8tz67TvMVlMt3StOTmULIpv0FS8/KmJqO53sKS5uSRkd+oiEBvjY0jMAAAAFwMZhrQxpkKU5sPNWruWD+V1ZoK9vNSTIRNb2XX6cHr/SVJr26rlcMuvfPvYerxr9mHkf19NO2PpZ5+Xsis0ehYH/31vlDZ/hVPg/289Kv3Ow4ODU1u7TjWqKSFJZ5jd4zx0yOTAiRJf9hYozEDfLT0gTBPn4m97Lr3tXJtPtSgGwf5KqGnXRn5jfr+tf5at79etyU7tbOwUWv31+vRSQHKLGjQ1GG+nscDAADAGpdNaOPvuc37B97cVaekhSVKWliijy4YOnSmSR+ebL7r0clSQwN72T2BQZJG9PORw+7l+f7jMlNj4xytLs7HDrB+d99h81KQ00sv/1uIJCk61KZnvxMsH1tz+/HzhsYMaN3nuH/NGLTMJNyW7KttRxtVVmsqs6BRM4f7aspQX63b36B9H7t0vtr0zEgAAACgc4QGtLEypzk0LH84zPPxl7uaL+JX5DQvUeoXbtORs006X/3JeqR9H7vU2OT2fN8/3KZdhY2tliztOt5oObaXlzThKoemJTv17qPhOlNpaNG6T2YmBvSwKft46z53Fjb3GRfZoNcrtQAAAvVJREFUnCymJTnV0OTWr9+vlumWbhzsq2nDndr3sUt/21mnUH9v9jIAAABcApYnoZWcIpcKSwwtnBGkCQNbX1iv2lOvd/fUa+GMQN17rb9W5tTrW38u1fyJAaptcOv5TTXy+tT5j0z0149XVuruJWX6ToqfCs40aUnWxf/NhTEDfPSDyQH6Y3qNbkh0aMJAh350c4AeXlqheYvLNGOEU6fKDL38Qa1SYnw0MbF538OwaLv6/2s51ZRhvvJ3eOnaBIdC/b21PLtOd4zxa7NhGwAAAB3j0gmtrMiuk81bur2duxx9N8WpijpTaXkNSuxt18r5YQoP8NZ/p1bp9e21+tWsIAU6P4kNd4zx0//NDtbJMkML3q7UrsJGLfpO0CXV88TNgRoZ46P/eKtCpTWmpg936tV7Q1VZ79bP3q3Sm7vqNGeMn5Y9GCbvTyWWaUnN9U/912e7t3TrvzZTT0v+cv7eBAAAwNeFV0VVtVuS3O5PlpW0fG2apuez2+2WYRgyDEOmYcgwDblcTXI1NsowDNU3NGh0yqgueArfHEeOfaSB8TFdXQYuAb8zAADwdcBMAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGCJ0AAAAADAEqEBAAAAgCVCQzcS4O+n8oqqri4DF6m8okoB/n5dXQYAAMAXRmjoRiLCQ1VyvpTg0A2UV1Sp5HypIsJDu7oUAACAL8ze1QXg4vn6OtS/bx9dKC1XyfnSri4HFgL8/dS/bx/5+jq6uhQAAIAvjNDQzfj6OhTVp2dXlwEAAIBvEJYnAQAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS4QGAAAAAJYIDQAAAAAsERoAAAAAWCI0AAAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS4QGAAAAAJYIDQAAAAAsERoAAAAAWCI0AAAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS/8fg6zMCYhZkJsAAAAASUVORK5CYII=)

#### Context Keys to Add

Using this table you can specify context keys depending on your experiment's [randomization units](https://docs.launchdarkly.com/home/creating-experiments/allocation#randomization-units).

## Advanced Event Settings

![Advanced Event Settings section with a LaunchDarkly event creation time dropdown set to Set to current time (the default selection).](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAvgAAAB3CAYAAAB2WzTHAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d1xXJX13f/xl3jweGQH46RHA+ngOOoBD00wMaiBxVKbOG1Tlpluo1ZrtuWye5m/2zlrt9lmq7Zcd8250pRNrciwJMOUFRSmlCAd9XAnqczQDsqZHoHD8fcHiKioWBl5fD8fDx95zve6ru/nXNc5Pt7X9/peV10O1nmP0eLYsea/BgKB1teBQIBAUxNNgQBNTU34/X6amppobGjA3+SnoaGRoYlDEBERERGRzhfS2QWIiIiIiMiXRwFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgogCvoiIiIhIEFHAFxEREREJIgr4IiIiIiJBRAFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgogCvoiIiIhIEFHAFxEREREJIobOLuBiV712AU+s9+K8dSZTh5jOuqzv/cU8vGovybfPZvyAr6jAL8FZ6w6UkzN3CaW+U9cyYBtzP9PSLV9RlWfio+qdNeQXf8Quj4/Qy6JwJKUzaoQTy5f47a/517M8U9idUT+fSnLEl7ddERERkfOlgP9FBKopq6gBoHKrC9+QRM4e8YOY1UnG0Kg2X6hQzP0v8N5oObkoGzCJh25LbPfLXP3WYp7J30t4bDLpQ8007i1n05tLqKrN5t6JjnMfr/b62JXH/P8twjL2fu68tvkExmAyY4kw0UO/KBEREelkF30c8e0qIP9jO6Out7UT1nxUvZWPu/8oMmIuQNjcs4WyGgNWazg1lWW4DieSGPbld3MxMFgcpF2f/PU6wQlUU7a5Cn+vNCbdnoktBCCVqCXzydlaSNloB8nmL6cry9WTmHb1l7MtERERkS/iIg/4HsoKiyiqKGRv4xSyR9rbBEwf7jcWs3R9FYb4SBJjkvmyJ4tUlbmoMTqY8F0LhUuKKP3IS+LVbRKjr4rCl3PZ+NGn+CPsJEaemMfieedZFrzqIfmnMxkfCwSqyX/iCTYaRjH9lxlYvW4K166npGIXHr+JPoNSGTMuA7sZ2FfAk08UYPp2JpHVRWz62IPBmkBG1kRSI1sO6SEX+a/mU7L9U3wGC/akDDJvSsTa0uwpzyfvzRIqP/Nh6O0gddTNZAwyn7Pu8+Oj9IX55Ox0MGnmJBJNQG0Rz/4hF0/KNGaOtYGviqLX8iks30Ud4cRcdQOZY5KJ7Abe4md55BUfyWMc1G0qwVXrxzIgjYkTM7DV5vPknwrYGwC25jBrjoupcybhPPWukhDA58XrA8IATCSMycZ8ECzGE8ep3RoOnN7HuO94WfOqGz/geWU+s7ZPYHZ2MjVrFrDwHROZM6aRFuEm99FnKYkaRWaYi8Kte6nrHkXyTZMYP6TlW9hQTdFLL1OwbS9+cwyJV5lxbSjDdts8JjmBWhf5r+RTUtl8/GKuSmvdLyIiIiJnc5HfZGsh+ZapjIo1ULV+KYvfcNMcRduE+9hR3H7Llx/uCVRRtq0GQ38HCQMTcFj8uMs/wtu6gA9X3hLytnqwXJVBmjOcvdur8B+vPN5JTIgHt6uq+Y0DbtwHoE98AlY8FP1jKXkVjcTcMJ7xN9hprMgn59VyTkRtP+53i/BEJpJ2rZ3QfaXkrd3S3H+gmoLnl1BQaSDuhtFkXBXO3ndWsiS/uX9/ZS6Ll2+kOjyRjJEZxIVUkr88h6LPzl33GTUepbbWg6flj9frB0w4nLGY6t1UftK8BV+li10BC454G+ChKGcxueU+7NeOZvTVVjybVrFkjftEf4G9lLxdRXhiOumDzHjKC8h7twYiEhlzWwZ2Ixhi0ph6a3rLCH0bIZEkDrNjOlTK8j89S+7GcqoO+TFcbsMea8PSjbPX0E4f8c5R3HqTE3MIRA2fwNTvxJ3xqoW/opBNjXZSRyQT46+i6JV8yhta9vGrS8j9oIbwwelkJFmpKS3FEzi+pofCFUso2GsmedxExqdY8Ly3iiWvu899HEREROSSd5GP4APdbGT8aCo8v4T89UtZHJhIOhtZuaEl3P8o48KMen5SRrnHgD09DpPBjGOQhcL3y/nIm9w87cPnoqTci8ExgakTkzEDyWEeHnn10+b1Ixw4+hlYW+mmBhumShd7A1Gkx1sBLzFpk8g2ReG40gzEwcdlrKpyszfgxN5SgvXaW5k6OhLwYNqzgNx/V/NpAMwfl1BUDfZxU5mQYgaGYfE9Sf4eNzUBK3veKaHGnET2D0djDwX6+9n71EZKKzykXl119rrPwF+ZxxOP5LW+NvRruRIxIIFYYznuHXthUBSVO6rw90zEaQM+KaJoRyP2myYxPsUCJGI+tICc8g9wj7UTBRBiJun72YwfZACfmU+35+D6dw1+kxN7fBSbQoDwKByDItv9Mlu/PYW7uq3h5Te3ULTGTdHrZqKGZjAxM5VI07lqmICjnT4i+lvIw4Cpr73l+LTPcGUGU25JwwLY6908saGGTw+As1clpVs9GGLHk31LKmYgtVcdv1nual4x4KGmxo8h2knq1YmYsdPHUkpVSCiNBMOPVkRERC6k4MgKJ4X8JSwBzAMuYLjHj/vDcjwhUST2acTzmYfwqEhMxS7KK7wkDzdDrYfaegN9bDEcj4AGY9vdbcE5OIq1+ZW4a1Mx7dyL35pKQl8AM5FW+Cg/h7xln+I96qOx3g8nZUkDptbtmTAYgYAfP+CtqcFHOJFR5tb2xFtnkggQqGLLPj/UlrB4bslJn8py0NuButtn6JfKxBscmI6PohutzVdNwhw4+hsor3RR3eDD9bEPs8NJjAG8/67BE/BTs2YBs9a03VgddfU0B3xC6X68f2PLTaz+xvMIuiYih09g2tBRVLvK2FJSSNGmXJ7xGrg3O5nQc9TwhW4q6N69dfXuJhMGfPgDgOdTahqhT/82+9gQemK9kBiGDbdRtn4VC/74AY4BsTi+lUzaWU4mRERERI4LjoAPJ4X8IlIvYLgH/Lsor/CA30PBX+ZT0KbJvbUM7/BUzC1B13CWSVCWuASi8tfiqijD9HEj1uEOIkNono///BIKGhyM//7t2HuB66WnyKv5MopvxN8ERKYyaXziSVOXQntGQb37nHW36xuROJztPZXGRJzTjuElNx9tbcTtNRPnjGnzxTPhGDOFDFvbgGvC8mXcrXu4mvKdn2Lo68DR10ykM5VIZyJRL8wnp+IDXLXJJFzoGtoTAmCArmf6+RmwjbyL+x1llH7owrW9kJx3Ctl00zTuTLdeoKJEREQkWARPwIeWkH8/qZgwXcCbEf0fl+M6ZMB27XjSY3u0vv/p+y+TX1lO2aFUUntaiDD6qfx3DX6sGKA5WLdldeDou5aCwvWE1lsYFm9rfv+Qm6oaP5aUdFIHRQJe3OdRn9lqJZxdVO/1wpVmwI/7zeUUeRPIvNmOtZcB9nhojLBh69m8jqe6BlOEAXwdqPs8mQc5sXddxaZ1HjxhcTj7N3/tzL0shIe48NRBVIyt+cvoq6H6qBVzCG3uZziTc3x9j7rZuCKPmvhJzLzt+CNMQzGZDEAj/sC5ayBwAX4il/XBavRT9vEuvNdbMQO+hsYT7YdcFBS6MSVkkDY2kbQx1eQ/9QQFpWXUfDsD60V+54yIiIhcWMEV8AG6mS7woxr97CovwxMSRep1yTgvP9HioIwiVxnl5R5Sr3WQ7LRQXrqWlVc0kmD2sGWjG/9J1VlJiO9D/ht78fdKI6Ffy9thVixh4P4gn9xwB6Z/l1JU6QeTv2M3WfZPZlhkCflvLiH3aCKW/5RTVLwLw7XphGMmMS2Jwr+XkPu3JdQMjcHkcVG0qY7EO6eTGdORutvZKx4XhRuPnrRUd9swkmNMYI7DcaUB104v5iQH9uMnX/1TSe1fSt47OTzTkEpCLz+7NhdRedkY7s9O7sAHNWEOA3/VJta+FUpyuvPk8Hv5MDISi1iyOZeFS/aSaAvH73FRutmLoV86jggg4uw1mNvro4cJc4ifvR+up8A0jNQkW0eOygndHCReZaH0vTUsWVlLQoQP13vl+I//HE0GvB8VUrDdg29EEpZ6N+7PwDTA0nplSERERORMFBfOl9/NBxXe5mkfl5/cZIhNwG7ys6u8HA8mHJmTyHSG4n5zJSvfrCQ8Nua0mGyNdxIVApZBjhNPgenmYMwto3CG7aUkv4Cyow6GxZuh3oPncAdqDIkk40dTybD5KVufx9oPfFjTbiV7VPMItWnQeO68NQNHSBVF+Wsp3AlxN00kI8YAHaz7NDXlFKzJI6/Nn+IdtS2NZhKcMRhCTNjjY0+cVYZYSbstm/FD++AtL2Dt+hJqLcOYOC6JDs02D7GRfEMiVv8uit514Tnt7MeE4wfTyL4pgfADpRTmr2VjeR2WxEyyp6Q1nwycq4b2+rAmkz4sCsO/t1BU8enneLKNAceYqYwfYsWzdSMFW2qwDrSd2C/d7GT+eBKplhqKXlnOyjdcEJfJlHGX8P9ITURERDqsy8E677HjL44da/5rIBBofR0IBAg0NdEUCNDU1ITf76epqYnGhgb8TX4aGhoZmjikc6oXuVg1ePE2mTG3JHbPxoUseN1L6s9mkhnTqZWJiIjIRS74puiIfO35qcp/hsU7rQwbZsdSX8Wmf1VB3wyS+p17bREREZGzUcAX+coZsF2XSdrhAjatz6MuEE6fARlMHTOKSP0iRURE5AvSFB0RERERkSCim2xFRERERIKIAr6IiIiISBBRwBcRERERCSIK+CIiIiIiQUQBX0REREQkiCjgi4iIiIgEEQV8EREREZEgooAvIiIiIhJEFPBFRERERIKIAr6IiIiISBBRwBcRERERCSIK+CIiIiIiQUQBX0REREQkiCjgi4iIiIgEEQV8EREREZEgooAvIiIiIhJEFPBFRERERIKIAr6IfHFNjTQ2dXYRIiIiAmDo7ALkPOwr4MkXfIy/LxPbhTw1C7jJXZCPeeo0Mvqe0laew5xiG/f/NBXzBSxBLgK123lzzZsUfrCdqtpGAEIjookfkszoMenER4R2coEiIiKXJgX8jvCXs+ThjcT8chppl3d2MWfgL2fJ3CWU1wMGE5a+sTjTRjN6iPWSOcjenUWUkUDqgM449fBR9f4mfP3TcFwOBGoo+MsSvDfdz/jYC9THoRIWP1NJ8i8m4TR9WX10zP73X+DPi9+h2jyIlOt+wE39wgnFz8E929lcvJpHi0r4TvadTLm611dbmIiIiFwy2e/SEGJn/IN3ktrTj/eTEvJWLWZ5YBpTky6NsXbPjiKKiOq8gF+ykRpzS/gOsZL8vYk0nnoF5Mvso2cco38QQ8RXHO6PfPgCjz5TQuh1d/LIrbFU5y7mb4u2c4hw4m+cws9+N5qdyxfyv888SUPog9z+rR5fbYEiIiKXuK4zH5z121PfPHbs2El/b/snEAg0/7epicCxAE1NASKv+FJTzNdPoIYPC6u4bHgyttOyig/3WzksXf4Sr7/1HjsP92LgwF4Y8VG65Hesa7qGb10RCngo/MsCNoWnE9/LT/nyueTuCrDj9WXkrCnE5b2cOIcVI4C/hpKXn2PpylzWFbv4zBTDwMgwQv7zMe99UEPjZ+/xUs6LrNu8l65XDsYWHtJS424s1wwlukcIxsuiiet9gPw39hE93M5lXcBbkc+ypTnkvlHI5l1+rAO+iaUbzVN//vwmO3au46WCg3xzeC/2FVdi/FYy3+zhoWTZk+RU9iIxrhehNeVs2HMZqd86zKuPPMf/xaQysCfgK2XJvJc4lHD6PvLtKWLVc8+xKq+QrdVduMJh47IuVeQ99jwfX3kN9nCapwX9YSlV/a/BbvZTs2U1y57L4cX1m6lqsOKItRDKmfabhZ0rf8+yzQc4tKeczTWXMdR5BaGnHKeqf63i+eWrWL3+PSq9l9F/gBVTl+Zt5pTu473Vq6gIu4bEyC5n6B/8NaWsXr6MlavzKSzbR2h0HNFh/0fe44t5e5+Xva7NVBnjSYz28vbSpey0XofDAgQ8lL+Ww5KcleS/U86+Y1dgj7mMUMBb/Cy/f+sAR0tzWfpiPu9VNtA33o6l7QcIuE/v47IdrHxmMz3SErAec5M7fymlB3ewfsU/WV1YTq25F4Etq1i87EXWvVvJYetABvYyNm/vsJuCfyxl2arXKaz4DNOVA4n6RgfmfdWXs+TxF6m+6g5+c3siXd9ZxMOr9xM//ha+56jn/dfW4b48gx98L4XIfRtY+ZaHb6ZfRV8NJYiIiHxldJPtF+SvWMOqDy2MmT6XufdlYt6WS8FOfwfWbGRvlY9hP3mQh6ZnYNpaQFF1c0v1mzms9w0j+8F5zP5JIkfWrWTjvpbVDlbj7ZvJvbNnM2VQLQUbyvCdoQeDzU6kt4q9XsBbwsqXq7BlPcjc2dNIo4g171SfWPhIHeHJdzJ7etv5/T7ca5ayviGNqeMcnDRQbLDjjPVR6Wrehv/jSqoudzSPLLflc7FmeRGhN0xj9pxppBlKWLmuCkJsJAxqxOWqaV5ujwt3NwcJfcH/yVpy3qxjyO2zeei+TKwVq1hTcXyftrffDDgnziT7GivWa7KZeUsipw5q+7fnk7MplIyfzWbe/ZOIrX6Z3E3e1m3W+aOYeN9sspNMZ+m/hqJV+dQ5p/Lg3NncnuAh/9USPCF2Mu+bRkY/M0m3zCQ7xXLasajZmMPL1TYm3j+P2T/PIHRzDrlbTxw5X9VeTCPuYvasuxjWWMT6LZ6TN9CBPgh48BjTyX7wIe4fY8W9ailFxlHcO/shpqUZKM0voXlveyl5cRUu6ximz5nNXUPrWP9SUUvb2R3ZspHiI4P4wcREegIHG01cPeY2fjZ6GCmjb2Osw8/Oik+AcK6e+F3ij5RQsOVIB7YsIiIiXxYF/C/IYLuB7J+Mxh4GmB04+vrweM4UudsKxTYsHXtPA4bLHdgvr8N7EAhUU+ZqxJmWjLUbGPqmkjkukajjR+oyJ6nDIjEZTNgHRmGo8+I9UxcGAz0MjfjqAVMcmT+bRHo/A4RYsA/og8fTJkT2dJAYb8HQOtLqp/a9HFZV2pjww1Ssp31TDNiddupcH1ETgF073YQ7ErCespR/ZyllPVO5wWnBYLCQmOqASjc1AbDFO2jc4aIGqHZVEhqfQGQI7CothyHfIdlqwGB2kJpoxr1z19n32znsqnBhHpqGM8IAYTbSx96M09LYus2owYlEhhngrP1bSJxwJxOHWzFgIHKAHYvXgzdwjs4D1ZSWekm4IQNbGBginIy61orrA1fryVnogGRSY8wYTJHYv2mm7uAZj+qZhVhxXGXDbDBgiXMQGWpjyHAbJoMBq8OOxVtLXQA4VMYHH0eSmm7HbDBgHZaKvdaF+1AH9uOOT+BKJ0Miml/bMrL52dhBzVdLakrZ+glExrRc0YtIZOiVjezasfv8P4uIiIh8brpw/kUZoeZfK8ndUYMP8NU2Yh30ObbT1U9z3DyCz2fCEnaiyRKfigVgHyefkoUAgUbOyO/jiD+USCNgMOH/eC3L/1lJrR8aD3ug/1nqadpLyfsGTOYk/Gc4DTTEJmB/eSOuA3bqKk04sk6N91Dn9dJYlceTc/Ob3whAo3lYc7Dt78B+ZD3uzxz43eD4XiTgx+ut49P3n2FOcctGAo0Qd5R2r4u07rez8eM97MMUfWJuviHSSXJL22nLnrF/A4YjLtY8V8reOj8E6vg0kHjO3qEOr8+M9bIT75gvC8fwHy8+aJ1KdPzHGEoonOuk4Vy6AiEQesov3A/g9VJX7yL393PIbXm/MWAlwgf0PNtGGzlyxAfmnpw2U+3AO/x5/j/ZGftDZmccv7HWRJg5lCNHNIIvIiLyVVLA/4Jq3skhzzOMO38+CYvBT+kLD1MKQCiEgP+8g1oPTCYfdYeBlukuvn1VeMNsp42On4vPXUl1WCTpZuCTtSwv9JP503txmMHzzrM8edaBVQvJk7JxbF1Mbp6Leyc6Tpv2gikWR/+XKfrXJvzEMj7y9K2YTN0JdUxk5tTTp81ADM4Bdax/twQaHIyKBDBgMpmIGjmNad8+dRpKR6Y+tceAOcyE77APjlfhrabKF4HNeuqjHM/Sf4OL3JxSzJOzufdKE1Tn8+QL5z69gHDMYV7qDtJ6TL0H6/B/IxbTF/hUn1uYiXBzEpn/NQHHef0LEEoPswmqDnGEk88F3AVred+YygN3XEvvrsffPcTBg430OP3GFREREbmANEXnC/L9xwcB8Df68OwqoWx3Y0uoN2DpaaZmhwuP34+3chNln3YgyoVE4nCEUv52CR4/+D8rJfeFPMoOn0dRAR+enYWszPsI67dTsYWA/z9H8QXA7/fh+8xNacWnNAbOEi67momwWHGOzcT+cS5rytubdmTCEW+jZlMJvgGOdp/NbxqQQOzeQvLLPfgB765C8t6pbunXQIzTjufdIurarG93xuF9L5/SfX4I+Kh+L5+Cneee9hQaGor/iBdfOydVMfF2vJsLcR0CfNUUrlxCwcftf/oz9t9YR91RIODH562hfLMbT9PxKwihGEL91HnbmVoTEkmC00T5xkKqfMAhFwXFNTiGtHPSdPZPeOY+zkeEA4flIwrXu/EGwF/romBNCTUdOBmNGRBL6O5SNh04+f3ew3/IfXd9l3hjmzcPbGPrbhMDBkR/sXpFRETkvGgEv6MCVeQ9+mvyjr8OiWLU9HvJuC4T54o8Fs7Px9w/kYT+fahq8AJmbNeNxrHsZRb8JheLPQFrRMd2t+07k7jhlZU8+z9r8JmiSPzORNL70jxF56w1usl95NfkGkxY+tpw3nA7o69uHoU2ONLI3L6SvCfm0xjhIHmAjT61R/DB2f+HVWFOxmS6eDIvl/L+k3Ce0mwalECssZpwR0z765sTmZjlYeWrC3n4n35MfRJI+35q6xfPYHPiMLsxJNhaVzEMGsPUA7m8/PeHWXnUQJ8BqYy/+dxj3ZHORCx/X878FyZy/9TEkz6XYdAYJtXkkvvUHHICZmzDxjNxmLndbZ6x/7AkRo+uZOWyBRSFWEkYasNmOsrResBkxpEYS+ErC3jWN407v31KbSOmMGrNKnJ+vxZftz44Uicx/qrzfb7lKX0MOc/VW1lJy5pI3cu5LJjjAXMMiSPHY+nA6X6PpHRSX3qC1f/cxLBpw1pG8RvZ+fqz/C8/YdHdx6cs1VH8z9eo6JnMA0kawRcREfkqdTlY5219Jubxx2MGAoHW14FAgEBTE02BAE1NTfj9fpqammhsaMDf5KehoZGhiZ87acjFrqGcnKdcJP7yfKd7yMXqyLZ/8PCfiyDph/ziR9cSaYQje7ezi2jio3pA/T42Pr+QpVsMpP/iv5gyWAFfRETkq6SAL5+f30vVv3JYefAGpt9s1+WgS8ihshf507Nv4g6N5urhiVx1ZS+6cYSDn2yn+L1Sqhq/yU133sUtCeGdXaqIiMglRwFfPicvJc8tIP8/idw8ZTzOsz59RYLSf3az8fV1vPvhdnYdqOMIJnr2upL4bw3jppuuxfaNzi5QRETk0qSALyIiIiISRPQUHRERERGRIKKALyIiIiISRBTwRURERESCiAK+iIiIiEgQ0ZMNO+hofQO1tYc4fNjXejOyiMipunTpQliYiYiInnQ3duvsckRE5BKkgN8BR+sb2LNnH717RdDHejkhIbrwISLtCwQCeL2H2bNnH/369VXIFxGRr5ySagfU1h6id68IevY0K9yLyFmFhITQs6eZ3r0iqK091NnliIjIJUhptQMOH/ZhNod1dhkichExm8M4fNjX2WWIiMglSAG/A44dO6aRexE5LyEhIbpfR0REOoVSq4iIiIhIEFHAFxEREREJIgr4IiIiIiJBRAFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgogCvoiIiIhIEFHAv5DqtrFi9m2MHB5HnHM4IyfPZNHb+zu7quBUsZAJqdksrersQkREREQ6lwL+BbOfFTOymVcRyx1PrOLVl55m1nWHWPqzbOa9X3/u1Zu2sTAzjtuWneGE4FztF7n616YTlzaH4qYOtvdLIevH40nq/ZWVKCIiIvK1ZOjsAoLW/rWsLu7OuL/OJSul+S37wEeZsS2N3+RsYMbVozB2boXBJTyJrDuTOrsKERERkU6nEfwLxWCkG4eo2dd2hD2cUTOXsfiupJZwX8eW56czIW0Icc7hjL1nIcW1QMVjjI0by2MV9RTNHk5c1iJ2t932Gdvr2LJ4OhOub97eyOx55Fed+WpBXdlSZk4eyRBnHEOuv405L7upB6hdQbZzLI9VnFh22+NjiZu6lP0AG2YyZGg2M2dNYLhzODPfhi3zRxKXNZM5U9MZ4ryb1UeAI+7mKUpD44gbPpLsP2xgd8uI+5b5I4m7exGrf3e8fSzT/7GNeupZfU8ccfespn7PUiYPGMLMDW2rPkN7xWOMdWazYj/AbhZlxXHb7xYxZ2rz5xueOZ2lZbspfvpuxqbGETcknQmzVrfWA7D/7YXcPW44Q5xDSM+aydKyDlxpEREREfma0Qj+hRIxmqk3L+Lu2ZO5230HU78/mpTYcIz9BnN8nHn3suncnRPOPY+u4vHL97B6/gNMnx1N3lMzWPXB9TyVNZnS77/B3yZHnzzaH99+u/v5n5O9qBs/nbeKx/vXUbxwJg/cUU/4i3NJCT+lvv2reeCux/De/CjLHo2HTYuYOffnzOv7KnMdHfh83mK2dZ3LEyuSsMfA7rehvmwLh2bOZ9l/24nusZ/8GbezsG4Kc3Meod9nG3hq1nQevPxVXsiOBqB+w1Jyfz6Lx3MeZH/+I0z/3WO8krGYrMc+YETGdIb/wcrTa2cx4qTajXyvvfaK00ssemMzIx56nFUz95O/4AHmTRmL/bp7mLFoBr33rGDOr3/DvGEjePrmcKhYyN0z1jJ45tO8mmSkImcOD/xyHtFr5jKiR4eOuFyk1q1bx7p16866zI033siNN9748A1LHwAABoBJREFUFVUkIiLyxWgE/4IJZ8TDq1j2YAre1+Yx+cbhDM+ayaLilhH9pi0sfW4bKb98iCkpdqIHjmDar7Lo+fbrFNeB0WikO4CxO8Z25vKc1t5UzIrnykn65aNMG2En2pZE1sNzGed/hb++cfo8/d2vrWCDZQoP3jeKwf2iGXzzLGZNtlKzY/dpy7arxyju+e8sUuLt9G4JwMaUO/h/P0ph8MDehFe+wl8LYpk27w5GDIzGnjKFGT92snltfuvVCOOwO3j0F6MYPHAwI27PIoVKtrkBoxGj0QgGMIa1++HP3t5i8KQZ3DFiMPb4EUybej3hfidZc+5gRLydwSPvIWtYPZXb3UA9G55bwv7vzmLWzUlE2wYz6lczGFe/lrUlGsUPducK7wr3IiJysdEI/gUVTtLkubwweRa7389n9XNP8VT2ZHY/vYq5g9249+yn+NfDifv1iTXqmwZTsx8IO8+u9rup2B9N4lVt7jI1JjF0cD0bXLuBk+8+3ePeidE+BXvX1oVJ+a8XSAGo3XLu/rp2x3jqt6crrVca6ivL2XakmDnXxzGndYF66nvHs//4tBij8cSVCaMRo+Eo9We4qfbzMHZtE/6NRrpjxtp6NcBIdyPU++uB3WzbXsfuHdkMWdlmAw2QcqAedLdE0Dse4E8dyVe4FxGRi5EC/oWy382W/UYGx0djxEj01d9j2tUpRN99Iw/8cwMz5tTT0NVO1h//wh2D2gTIrt3pecXn77b7qUe0CWjqhFFoPxAxirnLZ5DStiZjT6xdYdtXX9GZtZxUJN25mMe/3++kpu69T53bJMHq1JCvcC8iIhcrBfwLpO7dx8j+rZFH33qCUa0ZsTe9exlh71Hqe9uJDd9N5X4j0SOjW9errwdjV1pDZ4f1thPfezebt+6HgS2j9fVb2OyC2HT7aYv3sw+gfmU57qZRDG4ZxXe//Bj5YVlMu6Y7RuqoP9JmBf/5nSQYY6KxH97A/vpoomNbPx319cav33h4VyvR0UbcVfvpaUvh+OGqr69vd3qUBK+2gV7hXkRELlaag3+BhGdMJeuKfH4z/TFWF2/DXbmN4lVzeOy1elJGptC7awpZUwaz+Y+/Yt6r29i9x03xs3cz9taFuJuAruEYe8LuD4vZVlXHafH61PauKWRNsrPlT79h0dtudldtYfXcebzCOKZ+9/SHw0d/N4sRnhU88scNuPfsZttr85j5u3xqzFYIi8UZU0N+zgq27HCz5bXHeGSV+/x2QHwWU5J389dfz2TF+252V25hxa8nMPYPxad/lnYYw7sT7qlkS6mb/XXn335+whk1OQvrm//D9Kc3sK1qN9veeIzbR9/N0j1fdNtysdHIvYiIXOwU8C+UHinM+vsy7onaxl9nTGZs5gSmL6pk8H2LeeKW5hH7wXc/zeI7+lH8h8mMHD2BORut3PHwHS3z4qMZN3kc4QUPMHnWCk6/9fX0dvudT/P0JCOvzBrLyDF389T+FB5aNPeUp9C06P09Hn3mHvptmsOEG0cy+fEKkn77F2alGKHrYKb89h5it81j8rgJ/Oq5/SReN/g8d0A0WU8sZka8m4V3jWXkD37FiqZxzJ+W0rER/GFTmJq0m6d+PJnHNrVzSnCu9vNkTJnF3/6YBWseYMLokWT/cRvx/zWXrH7nXldERETk66TLwTrvseMvjh1r/msgEGh9HQgECDQ10RQI0NTUhN/vp6mpicaGBvxNfhoaGhmaOKRzqv+K7HRXMcBu6+wyROQio387RESkM2gEX0REREQkiCjgi4iIiIgEEQV8EREREZEgooAvIiIiIhJEFPBFRERERIKIAr6IiIiISBBRwBcRERERCSIK+CIiIiIiQUQBX0REREQkiCjgi4iIiIgEEQX8DujSpQuBQKCzyxCRi0ggEKBLly6dXYaIiFyCFPA7ICzMhNd7uLPLEJGLiNd7mLAwU2eXISIilyAF/A6IiOjJ/gO1HDrk1Ui+iJxVIBDg0CEv+w/UEhHRs7PLERGRS5Chswu4GHQ3dqNfv77U1h5i/4Fajh071tklicjXVJcuXQgLM9GvX1+6G7t1djkiInIJUsDvoO7GblzRt3dnlyEiIiIiclaaoiMiIiIiEkQU8EVEREREgogCvoiIiIhIEFHAFxEREREJIgr4IiIiIiJBRAFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgsj/ByliDju321arAAAAAElFTkSuQmCC)

### LaunchDarkly event creation time

In this section you can specify how to derive the creation time of the event, which populates the `creationDate` in LaunchDarkly payload.

The available options are:

1. **Set to current time** (default): sets the event time to the current timestamp.
2. **Set from event**: sets the event time from a client event property.

#### Event property name

![Advanced Event Settings section with LaunchDarkly event creation time set to Set from event, revealing an Event property name field pre-filled with x-sp-dvce\_created\_tstamp.](/assets/images/07-advanced-time-from-event-32eeb82cf4abf42f8829f122c934cf6d.png)

This text box is revealed if the [LaunchDarkly event creation time](#launchdarkly-event-creation-time) is set to "Set from event". Here you can specify the event property to use in order to set the event time (in Unix milliseconds). For example in the above image, the LaunchDarkly creation date will be set from the device created timestamp (`dvce_created_tstamp`) of the Snowplow event (prefixed with `x-sp-` in the client common event).

## Versioning

![Versioning section with a Version text field containing the example value Container Version.](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAuwAAAB7CAYAAAAv3NUuAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABzPSURBVHic7d1/dNT1ne/xJ+E7hG9wAgx00Iw6oRllYCfIoIkl1MQa1uCa6riSrXBrvMZT7Gnale3l9lp7rddy+uN02V27p/S2nCvtgsfYQpeoQRNrqKQaKiihJsJQJivBJoQRB8hIRjLfmdw/EiSEYFEGM+DrcU7+yPf7nc/3/f1M4LzmM+/5zph3Dx/pB+jv7yeZTJJIJEgkEvQnE1iJBJaVIN7Xh5VI0NfXx7X+OYiIiIiIyCcjY7QLEBERERGRM1NgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwC4iIiIiksYU2EVERERE0pgCu4iIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwP5hrCAbVnyLb/2wllDy1F2xnWt55FvfYmV9V+rPGwuy4ceP8JNNHVipGG9fAz959Eds2BVLxWgiIiIi8glSYP8whgef1w5HQwT/MnSHRXuwnViGE++snNSfd6yBfbID+yU2jFSMN84cGC/TlorRREREROQTdEEF9ti+Rmp/38HI68QxOn5fS+O+VK4iG3hmerATpj04ZCW9L0Rwbwymesm/PIWnO2Gch7KvPEBVSYpeDOQUU/WPSynLS0n8FxEREZFP0AWU4CK0NjXTvKuJzvjdVN3swfxgX4zQC2tYt7kDY1YO/txCHCk6q3F1Ph6zhdbgbsI35+AErI4gu4+Bc24+7sGXPFZ3C/XPNdLyVgQry4X38+UEbnBjJjuoW7mK5kv8FNo62NblpvLhxXiTYVo21dK4cx8Ry8Qx3c/C2xfim2JArIU1K2rouLaKR+/0DozftY26TU207h8c/3NllH/Bgx2Ibl3ND5+OUXirl57t2wgetnBcVUxFRSluE9izgUce30HeXQ9TOddG25PfZW1HPmVF0PZyKwf7ssktDLD4Vi/2E/P5+/VsejnIwaQDz/VeeLWJ8NylPPhFT4pmVkRERETOxgW0wu6g8K5KyvIMOjavY80LocGV9iFhPa+M++5KXVgHYFwe+VeZWN1Bgu8ObNoXDBLFgfdv3AMbom2sf7yG5iNOCm9eSNEVMVo3raV2Z/SDYax9rQSTbgqvy8MBdL24lppXe3AWB6j4Oz/ZnU08uW4LXcnTS+BoC08+voFthxz4b1pI8XSLYMMa1v5+yKp/spNtL3eQ7S+hZIadSFsjdX8Mn/m6DrfQ/KYN3+dLmTslRugPtWzZN3g5O9ZT09BGz5RCSm/MJzu0neCxc5lEEREREfm4LqAVdmCcm9J7KuE/1tKweR1rkhWUsIX1Lw2G9XtKyRmX6pOa5Pm8mDtbCf45SvH1EYJ7IuAoJv/KgSMiO7bQGnVR8t8qKM2xwXVOYm+vYdsbIQKzB18+5JRS+ZVScjIALFoORcDMZ+7nCvGZ4HY4aAtnY4wQ2MM7mggec1L01UrKcw3Ah3l0JbXNzQRvWIQLIMPO3L+vIjDDgJidg3tqCB4IY+Ec+UnO9FF+7yL8Jlg5h9n9+A66DkQhF1pfDxKd4Kfy3gA+E5htJ/LPtURSPbUiIiIi8lddWIEdhoX2tawF7Fedr7A+wLzKS57ZQnDPbqJXHKb9EDjmewfbYSy6DoSxkjEaf/YIjUMeZ9gjRAfX+40JJvYP3s8w8BYU4tzTzJMrw3i8eXhmzKVgfg5mBhAfenaL8IGDWJn55F5+4ulykDt9GnQcJHyUgcCOjfGZg/szTbIMwIoT5wxPcoYNW+ZgNZnjMYG4FYdklMghC+NS90A7DYBhXIB/KCIiIiIXhwszhw0J7c0UndewDoDpJf8qk7a9QZpbDnMQB4WzhvRyJ+NguCm9pxxv5tA6HTjOsC5tzgiw7Js+drzeRvDPbdRvb2KLbxHVlWdu6Tn1yRqM4olzurKRjQXGXph/GiIiIiIXmwuoh32YcW5K71nO8vMd1gEw8c7KwzweZMvWTqyJXnzuE/sMnM5pkIwQOe7EnevGnevGlWXDdNpHfkWUDNNSX0d9exaFCwJUfm05VfMcRIM7CR4dfrCB87JpGMc7CXWduCt7lH0dETCn4Uxpwz6Q4cAx2cDqbKfjxA13rPdTcz94EREREfnILuxl1HHmkDvFnF/mDC95tjbajoPD6yN3yMw5C4rxba2hZeNq6PbjopPWra1Yxct4oGSEwTJMiLTQtLWd948V47VHaX2rB+xenBMY1hIDzrnz8L68gebfPAnXezC7t9PUDq6bivAYnOE2lx+XnfxrvTT+poWNa00Oek0OtzYTskjth3lFRERE5Kxc2IH9kzRhJt48g7ZgNl5f7qkTN9HPkvvi1NU30fqHelrHOcidE+CO+U6gY4TB7PjvuI/Yc3U0v7SeHcdtONxzCfxDGR6D0wI7EwupuNdi06Ymtr8QhCwX3gWVlN+Uc16eQPvcChYfhU0vb6Ox24FnthdXVwsx9MVLIiIiIp+0Me8ePtIP0N/fTzKZJJFIkEgk6E8msBIJLCtBvK8PK5Ggr6+Pa/1zRrtmOe9iRI/asE8cfDmwv46VP2vGLFtG9Reco1uaiIiIyKeMVtjlNLG2Wh7bGMZzfQHuCVFCW5sJZ+YSmK2wLiIiIvJJU2CX05jeUu4oqKfx9QZao5B9eT7ld5RTNGW0KxMRERH59FFLjIiIiIhIGrtwb+soIiIiIvIpoMAuIiIiIpLGFNhFRERERNKYAruIiIiISBpTYBcRERERSWMK7CIiIiIiaUyBXUREREQkjSmwi4iIiIikMQV2EREREZE0psAuIiIiIpLGFNhFRERERNKYAruIiIiISBpTYBcRERERSWMK7CIiIiIiaUyBXUREREQkjSmwi4iIiIikMQV2EREREZE0psD+aRSPj3YFIiIiInKWjNEuQD4Zve2v8Ex9M68F/4t3eoEMGxMv+yzXXL+A2xb4+EzmaFcoIiIiIiPRCvspLNqeWsHKTR2nbk6GqPvxCjbsslJzmmSYxp+upLY9NcN9uB52/faf+R8/eoLmI1OYF6jiG9VL+cb9X2LB1TZCm1bx7f+zhi2dWnUXERERSUcK7Kcw8M6eSWxPkK7kkM37gwQTM/FdnaI3JDKcFN5WQfEVqRnuzOKE1v+Uf/1dD7PufoR/eeBGbG8+w8//72p+vq6Zo94v8b0fVnPLJW2s+ZdfsvXQ+a5HRERERD4qtcQMY1ztY+bTDbR2l5GTM7Ct480gljeAxwCwCO+oo7ahhX19djzzAiy+2YOZDFG7spZIjknnXiiprqZ4apiWp9fTsLOTHmMa+TcGCNzgxiTMtmdqiN7yIIE8IBmh7fla6l8LETWm4f18gECJGxOLtidX0Gwvxt7eTOu74LruDipv92E/m4vZ9wyPv3CIWUse4hvFWby2+t/47f4rua3qNiYGf8dT/++XTH3kf3LnP91P/AePsfbX25lVXcDE8za7IiIiIvJRaYV9OMPDHG+M4K7wwO/JLoJ7LLyzczEAa389NS/2MOe+h/neN8tx7trAphOtMokosallLPtONcVOiO2spy7spfI7P+B7Xykg1lRLc/fppwxvqWFjl5uK5T/g4a+VYnu9hto3YoN743R2xCi499t8b1kp5huNNHedzYXE+dPvt/HO5X/LXV+YCvQSz5rBbUvu5s55BSz48m1cZ3ub3Xt7IWsGtwUKoPUlmrXKLiIiIpJWFNhPY5A720s82EoYoLuVYNKHf/rAmxH7WtpgzgIKnQaG3UuR305o776Bh4514p3jwT5ucKgMIBYlEolhXFpE5bIqipzDTpfsoqUlSv5NpbgngDHZR9l8J8GdQQYiuw13QQmeiQbGFC+eKT1Ej5zNdXQTequHz/h8DLxRMJV5X67izrnZABxt3UkoPgX3lVkAZOXnMyvjbUL71MsuIiIikk7UEjMCY7ofb18dwXdLsXYFifsCuDMALKLRHg6+9gse2Tp4cDIOM9/HYvxp45hzKqg81kDjUyvZ2DeNmfPLuHX+8GaWHqIxO85JJ7fYJ2VjvBclxgjGWpxVpE7E6e21MWnS6Q0uvTueYMXqHXwm8E/clnviorPJGg+HonHAdjZnEBEREZFPgAL7SDLc5M+waGhrIx6E/IB7cIeBaZq4bq6m+gbHqY9Jhk4bJnb4MPY5AarmB7DC21j/qxoaXQ9Snjv0qGzsE6L0HAGmDGyJHunBuiQP81yuYWwWE7Pi7IocBbKH7DjElk2v0HfNUr6x8IqT0fx4D0d7IcuusC4iIiKSTtQScwbua7zEXq1je9JHfs7J7R7fTKKvNtDSbUEyRterDTTuHXEtnMPb17N64zYiFhgTsjENsBLDDsrIId9n0raliY4YcDRI49Yw3jnecwvsTMGTl01Xawsdp5wzi2v+vpp/rPCRNWRrb2sLe7mCmXkK7CIiIiLpRIH9THLy8Y7rwczPJ2fILBkzbqVyHjT/cgUPPbqS9XsNPDkjR+ucGysoNraz+vsP8dC/1hHxBSjNG+m4uymbEqTmxw/xyM8aiPkXE5h9bnEdbMz6wnzc4Zd4qnHIJ10Tb9P4y9X89k9DXmT0tlFT24LtmgUUTT7H04qIiIhISo159/CRfoD+/n6SySSJRIJEIkF/MoGVSGBZCeJ9fViJBH19fVzrnzPaNctZi9Px7GOseOYQsyru56s3f5Yseuna+zbxqTNwTwYOtbBu1TpejPr4xnequE6BXURERCStKLBf9HoJbVrNv9fuIX6Zj6ICH1c6sxh3/Chdf25h62v/xRHnfO77+peY51Q7jIiIiEi6UWD/lOjt3M6Lz7/Ca8H9dB2NgS2bSa48rvtcCbcUz2CisrqIiIhIWlJgFxERERFJY/rQqYiIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNGaNdQDqKRt8j+l4viWSS92PHR7sckU+1rKzx9Pa+P9plnGbMmDFMmGAyefJExmeOG+1yRETkIqbAPkxPz3tE3zuGaZo4JmePdjkin3p7Qx1c5XGPdhmnSSaTRKPHePvtA1xxxWUK7SIict6oJWaY9471kpWlsC4iHy4jI4OJE+04P+Pg8OGjo12OiIhcxBTYh0kkk0yepLAuImfHbp/AsWOx0S5DREQuYgrsw6hnXUQ+ioyMDPr7+0e7DBERuYgpsIuIiIiIpDEFdhERERGRNKbALiIiIiKSxhTYRURERETSmAK7iIiIiEgaU2AXEREREUljCuwiIiIiImlMgV1EREREJI0psKdYrKuNti5ryBaL8K42OqKjVpKco9hf2mjrHvpNlgPPaeexUStJREREPkWM0S7gYhLZsZbVm6L4l+ThwwBiBDeuoqbDQ8W9voGDrAhtv6ujcUc7B6NxTKcH/4IAC2c7RuXJiLyymp+86mbpsjJc5/HlW3Tran64ySDw7SoKJwzdYxF8agVruotZvqwUZ8rOGKPliZU0mBUsv9N7TnNrvbePLevraL29msWz7UCc2DvNrN24jdL7qyiamqqa5Zz1H2D701tof3/w97GXUXh7CZ8dB8T+zOa6HYQTg/vMq7nptrk4x0D4tWfYHOod3GFwWcFtlOSN++TrFxERGYFW2FMl2UHT80EcCyopyzMHtnU30bDdRvGSAL6JAFFanlzFkzvBd3sVy5bdzx1zoO2p1dTuin3Y6GfN2rmGh1ZsIJQ8u+OzpxdQPM+L4zz/Jdh9PnKTIdqGX6cVom1PDJcvP4VhHcDEPaeIor9xnfMLIbu3nMob7bQ+v4WO5ODYJXdTfkUHDS8Gsf7aAPKJSvTFiMVO/CRO7kgm6IsN2dfXx4m9Cat3yGN66UskRhxbRERkNCiwp0q0k85jDtxu+webrO6DHMx04RpcgbXaG2nYZVJyTyWlPjfOS934bqqk3Bdjx0stjEbXjJHjp3SeG/N8n8iej286hN4MckpzSaiN3TEXPl9q4zqAw1dKsdf+1w88C3a3m+yjXRz8oHgT9+UO4t2dRFJyBjkXfT1hwgfDhLvfpfeUrB3jyMGBfQfeeY++obsSvRzpDhM+eIB3e08f78DBMOGDR4j1n//6RUREPoxaYlIlObjOmjFsm2F8MMmdu4JELi3AnzP0gQbeolLmtprEADsQ3dXA+vpthMIxbFM9FP1dBWWzBoJnx9M/4heHCgg4gmx+vYMemxPvgsUsmT+N1l89RM2ugVFXP9iKv/JhFvsMIrsaqK3fRigchYlu5i6oIFDgxAAiW1ax8nUP1d8swxXbxuoVTThv9dPzahPBcBzzyiIq7i7nRO6NBhtY/1wzoXAcMyefkkCA4itNiG1j9YoGjDluIruDWHPv58Evuodcp518Xy51m1oJxvz4TQCL0Bu7iV5aRP6lA0dZh1qo+20DLR0RsLvJ/6DWMI3//hgtk+aSfWAH+yYsZPnXizH3N1H7n1to7Y6C3UX+jYuomO/CwKLlV9+lzl7Fw3d6Bmr/WPM6uEKfARDHGrqcPtaApEX84/3FSAq9++Zm6oMjvEuVCPOnFzfxp5EeFOvgj/UdI+xIEG7dTH0rMPZy5lf8LVef91e0IiIiZ6YV9lRJxLGwYQyZ0XjcggywAWAROdyD4XDgGPZQI7eYRV/04wSs/XX84okWjILFLPtfy7i7ALY98Qvq9p9MitbeZnaOL2Hx1x+gstCkfdMmth018H/5Bzx6pxdjgp+qRx+mwmdAVwNrnmjBmFfJ8m8/yP3FdnZvrGFL+AzXkQyzbWsYzxeXUv3VAJ6jTax/MTRw3r80sKamlewbqli2vJpyd5jGdbUEj594bJSOcDal91RTdaPrtKEH2mLaad0zGKysfbTtiZ5sh4kFqX28ls6ccpZ+czlVNznY9/Ra6vd/cOWE34rgvvV+qu8qwGGF2LSunvD0O6he/iD3L3DSsWkNde2nN6l8/Hkd3Jlhw4BT2l9sBpAc/JELxFjMiZOwqz1dREQuIArsKWERfiPIwQku3CfSeDJC6+5ObC43zhOznAQM24eME6P1pe1EZ5VRcYMH52QnnpLFlHujbH/5ZK+0cXkJi27x4b7UhfemItwcpCsMGAa2TAbCZebgyv7kAhZ/fSlL5rlxTHTgnlfIzMyDdHadofM6w87c8gqKrnLhyi2keLaD2IEuolgEm5rpmR0gUODGOdWF/9Yy8q0gLScCcoad/LIA/lwXTvsIb97Y85kzPU57WzsxwHprJ7uPufDNHmiHie5sYoetiMCtPlxTnbivD1DqidD2pxOroAbOebdS5nPjmmrC8QiRYyauGV5cUx24rw+weFEpntNWQ89xXgGmuHGZYYJvhD843n6Fm+wjwWF3BZK0lmEnb/4CZk8dO9qViIiInDW1xKRAx7MrWfWHGL4llXgMgDCNP11JQ9hN+TL/B/3hNgOwPqSBIhmhMxzHef3QnnITt9tJ/PVOIgzeaSZz/Mn9Y20YGRbxMw1rZmPbU8/a37TRcaiHOGD1WXjPWIQNY/zJPwubzYAkxInQ2R0jGl7Dd18/ebRlWXiOxk8+9sNej2Bn5uxcap/bQftxL+PbdhPNKSJ/sH39YNdBrHCIVf97y8nxkxaGL4o1eMW2jCEnmJDPTfO2sm7tD+n0zsQ7w0fBnCLcmXDKWngq5jXTS/ntXh57YiWPvfcAy29xwZULWXTtSlb/dAXR//4oi848qSIiIiIfmwJ7CrhuvJvFsRrWb26mc3YZrgwnRXdVEnvqSZr/EKLoDg8GBvaJ2VhvRYjAKXdEsfZvoz6YRdGCyYPtMyOn3o/TK23tq2PNhg68d1WxeJYTkxAbvr+Gno86UDIOSQPXDVUsvv7UW1Da7OZZt4XYZ80h99lNtO4OYe6K4ZrnP2UujOnlVN/pO/VDsJnZGCN+tNPE88UHeHBeiOCbQVpfXc/KF10E7q+icNitFs95XpOdNL/Yinn9YhYXTxvYdqiZhp3gv6OKMs/ZDCKjZaztxIr6WMZmGDB2LGNP/DkkEiTU1iQiImlMLTEpYNhd+Iv9TDsUon0wV5pOHyUFufT8OUjnYBhweT3Yu9to6R76aItQcwPNoQhkTMPptBF+u2PInVRidHSEsTldTPsYtUVCHfQ4fRT7nJgZQPx94h+ngyPDgXMqRN6JYp/qwDH4k23Pxp75Ecaxz2SOO06woZbWY9PwzT7Z0e9wOuBQmKj95PiOSdlkj9ReA9DdQsPzLUQdHvwl5VR+/X5KzBBNO4Y16KdiXsNBgoemUVjixzVhoJ7o3iCdWT6KCtycqURJA2MvY+6tFdy16C7uWrSQa6aa5BWf+L2Ckun6RKmIiKQ3xYxUybRhECM2JAybmQZYJ+8iYlx1E2VX/4TaJ2owy4vxTrII79pC3Rsw98sFODDILvbTsLqO9a+MZ+GsbHra6qkL2ilYenZf/mOMM7HFDtL+VphpOU7sUydj625h86tuCqa9T+ilOlpiFt7jcT7a02+SP7+QhsdrWfeCjVvn5kBXM7XPduH/2lKKxp/tOHZmzvZQ+9sgscvL8A9ZCXfMKcL70npqf+0kcLMPZ1+Izf+5md4bH6ByzghDZcboeKWBEDYCBTnYDrXQEbXhmDw8gBnkn+O8YsWIYZzS8hPvi4Pt1A8aSxpKHGB77VNsB8iYhG/hAibu3MgrXbrXuoiIXBgU2FNl6AdLh0oOXc52ULi4Gp6rZfOvV1EXA/ulHvx3VbNw1kDINHIDLF1Sx4b6dTz2bByb00PRkirKcs/yqcorovjytTQ+vorIkodZPOcOKt6uoe7Z1Wwb68BbWETxjG2E+6LwEe++blxVztIlBhueX8+qzXFsk3Px3xKgcDLwEb73yT7Lh+fpILFr/KfeMcfuZ8l9cWprN1PzWB1x04mnMEBgtgkj3aV+chGLvxxl43MbWbUlCqYTT+FiKq61w7CvMzrneU0C2E59TyphQYZxhkYbERERkdQY8+7hI/0A/f39JJNJEokEiUSC/mQCK5HAshLE+/qwEgn6+vq41j/SUufFY2+og6s87r9+4HBWkJrv1xC/42EqZw+GwP11/OjnXRR/eylFqfn+Hhklse1rWFGfTdV3FuHJALAIPrWCtX0BHq70n/8vnvoUO5t/kwe2PjXyfdiH+6gr7Gd5H/aP/f+GiIjIWdCb+alieCie56D9hVpawoOru1cWUTy9k6aNzXR+hBVoSS9WVzPrX+zANX/eB2E98kYtdbvtFBXnK6xfSJJH2P27jfyxW+0wIiJy4VBLTMoYuG5eytKpTQQPxcBpBxwU3VtN9ksthI+BS8nughQ7FGPaLdUsmXPifjZxIofHU/yVagov1z+hdDDpqs9z02XnIYSPGYdDX7IkIiKjTGkjpUxcc8s45Ts+DSe+BWWjVZCkgH12Kac+gyaekvJRqkZGYk69HPfUv36ciIjIhUgtMSIiIiIiaUyBXUREREQkjSmwi4iIiIikMQV2EREREZE0psAuIiIiIpLGFNhFRERERNKYAruIiIiISBpTYB9mvJk52iWIyAUkmUwyZsyY0S5DREQuYgrsw4zNyCByuGe0yxCRC0Q0eowJE/Q1xiIicv4osA9jvySLWCzG4SMK7SJyZslkkqNHo4TfiTB58sTRLkdERC5ixmgXkG7s9kvo74f3jvXy3rFe3o8dH+2SRD7VsrLGszfUMdplnGbMmDFMmGByxRWXMT5z3GiXIyIiFzEF9hFkZ19CdvYlo12GiIiIiIhaYkRERERE0pkCu4iIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwC4iIiIiksYU2EVERERE0pgCu4iIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwC4iIiIiksb+P++HVVRtwz/DAAAAAElFTkSuQmCC)

The LaunchDarkly event import REST API accepts a `User-Agent` header, which helps identify the source of traffic and debug issues. One of the components to construct this header is the `Version`, which can be any format.

### Version

This configuration section allows you to define the version to be used. For example, you could use the `Container Version` as the version to use. If this is not provided, the Tag will use the string `"1"` instead.

As an example using the default version, the User-Agent header could be like:

```text
'User-Agent: MetricImport-Snowplow-int/1'
```

## Logs Settings

![Logs Settings section with three radio button options: Do not log, Log to console during debug and preview (selected by default), and Always log to console.](/assets/images/09-logs-settings-9d0b55515c253bf74027121c7524c6a0.png)

Through the Logs Settings you can control the logging behavior of the LaunchDarkly Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the LaunchDarkly GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{

    "Name": "LaunchDarkly Metric Events", // the name of the tag

    "Type": "Message",   // the type of log (one of "Message", "Request", "Response")

    "TraceId": "xxx",    // the "trace-id" header if exists

    "EventName": "xxx"   // the name of the event the tag fired at

}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | "Message"                                                      |
| Request     | "RequestMethod", "RequestUrl", "RequestHeaders", "RequestBody" |
| Response    | "ResponseStatusCode", "ResponseHeaders", "ResponseBody"        |

---

# Snowplow Client for GTM Server Side
> Receive Snowplow events in GTM Server Side containers with the Snowplow Client, which populates common event data and rich Snowplow properties for tags.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/

To receive events in your GTM SS container, the Snowplow Client must be installed. This works for both events direct from the tracker, or enriched events from the pipeline.

The Snowplow Client populates the common event data so many GTM SS tags will just work, however it also populates a set of additional properties to ensure the rich Snowplow event data is available to Tags which wish to take advantage of this, such as the Snowplow Authored Tags.

## Template Installation

There are two methods to install the Snowplow Client.

### Tag Manager Gallery

**Coming Soon.** The Gallery for Clients has not yet been made public.

### Manual Installation

1. Download [template.tpl](https://raw.githubusercontent.com/snowplow/snowplow-gtm-server-side-client/main/template.tpl) - Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select "Save Link As..."
2. Create a new Client in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

![Animated walkthrough of installing the Snowplow Client template in GTM SS: creating a new Client in the Templates section, clicking More Actions, selecting Import, importing template.tpl, and clicking Save](/assets/images/manualclientinstall-93c4a4f45c4decdd25c20faf1d519013.gif)

## Snowplow Client Setup

With the template installed, you can now add the Snowplow Client to your GTM SS Container.

1. From the Clients tab, select "New", then select the Snowplow Client as your Client Configuration
2. Click Save

![Animated walkthrough of adding the Snowplow Client to a GTM SS container: navigating to the Clients tab, clicking New, selecting Snowplow Client as the Client Configuration, and clicking Save](/assets/images/clientsetup-a7275d01ad3c7e318d302a322af2c3f5.gif)

## Testing

You can test your Snowplow Client setup by using GTM SS [preview mode](https://developers.google.com/tag-platform/tag-manager/server-side/debug).

Once your debug container is running, obtain your preview header value:

![GTM SS preview mode showing the container debug panel with the option to open the “Send requests manually” popup to obtain the preview header value](/assets/images/preview-mode-1-965b75ae77d43b5fa941eaed2e378f5e.png)

![The “Send requests manually” dialog in GTM SS preview mode showing the preview header value field to copy for use in cURL requests](/assets/images/preview-mode-2-8913f55547ae5b9dbf3debea2c2ac704.png)

Now you can use the cURL command below. Note:

- Replace `{{your-gtm-ss-url}}` with the URL of your GTM SS container.
- Replace `{{your-preview-header}}` with the value obtained above.

```bash
curl --request POST \

  --url https://{{your-gtm-ss-url}}/com.snowplowanalytics.snowplow/enriched \

  --header 'Content-Type: application/json' \

  --header 'x-gtm-server-preview: {{your-preview-header}}' \

  --data '{

  "app_id": "example-website",

  "platform": "web",

  "etl_tstamp": "2021-11-26T00:01:25.292Z",

  "collector_tstamp": "2021-11-20T00:02:05Z",

  "dvce_created_tstamp": "2021-11-20T00:03:57.885Z",

  "event": "unstruct",

  "event_id": "c6ef3124-b53a-4b13-a233-0088f79dcbcb",

  "txn_id": null,

  "name_tracker": "sp1",

  "v_tracker": "js-3.1.6",

  "v_collector": "ssc-2.3.0-stdout$",

  "v_etl": "snowplow-micro-1.1.2-common-2.0.1",

  "user_id": "jon.doe@email.com",

  "user_ipaddress": "92.231.54.234",

  "user_fingerprint": null,

  "domain_userid": "de81d764-990c-4fdc-a37e-adf526909ea6",

  "domain_sessionidx": 3,

  "network_userid": "ecdff4d0-9175-40ac-a8bb-325c49733607",

  "geo_country": "US",

  "geo_region": "CA",

  "geo_city": "San Francisco",

  "geo_zipcode": "94109",

  "geo_latitude": 37.443604,

  "geo_longitude": -122.4124,

  "geo_location": "37.443604,-122.4124",

  "geo_region_name": "San Francisco",

  "ip_isp": "AT&T",

  "ip_organization": "AT&T",

  "ip_domain": "att.com",

  "ip_netspeed": "Cable/DSL",

  "page_url": "https://snowplowanalytics.com/use-cases/",

  "page_title": "Snowplow Analytics",

  "page_referrer": null,

  "page_urlscheme": "https",

  "page_urlhost": "snowplowanalytics.com",

  "page_urlport": 443,

  "page_urlpath": "/use-cases/",

  "page_urlquery": "",

  "page_urlfragment": "",

  "refr_urlscheme": null,

  "refr_urlhost": null,

  "refr_urlport": null,

  "refr_urlpath": null,

  "refr_urlquery": null,

  "refr_urlfragment": null,

  "refr_medium": null,

  "refr_source": null,

  "refr_term": null,

  "mkt_medium": null,

  "mkt_source": null,

  "mkt_term": null,

  "mkt_content": null,

  "mkt_campaign": null,

  "contexts_org_w3_performance_timing_1": [

    {

      "navigationStart": 1415358089861,

      "unloadEventStart": 1415358090270,

      "unloadEventEnd": 1415358090287,

      "redirectStart": 0,

      "redirectEnd": 0,

      "fetchStart": 1415358089870,

      "domainLookupStart": 1415358090102,

      "domainLookupEnd": 1415358090102,

      "connectStart": 1415358090103,

      "connectEnd": 1415358090183,

      "requestStart": 1415358090183,

      "responseStart": 1415358090265,

      "responseEnd": 1415358090265,

      "domLoading": 1415358090270,

      "domInteractive": 1415358090886,

      "domContentLoadedEventStart": 1415358090968,

      "domContentLoadedEventEnd": 1415358091309,

      "domComplete": 0,

      "loadEventStart": 0,

      "loadEventEnd": 0

    }

  ],

  "se_category": null,

  "se_action": null,

  "se_label": null,

  "se_property": null,

  "se_value": null,

  "unstruct_event_com_snowplowanalytics_snowplow_link_click_1": {

    "targetUrl": "http://www.example.com",

    "elementClasses": [

      "foreground"

    ],

    "elementId": "exampleLink"

  },

  "tr_orderid": null,

  "tr_affiliation": null,

  "tr_total": null,

  "tr_tax": null,

  "tr_shipping": null,

  "tr_city": null,

  "tr_state": null,

  "tr_country": null,

  "ti_orderid": null,

  "ti_sku": null,

  "ti_name": null,

  "ti_category": null,

  "ti_price": null,

  "ti_quantity": null,

  "pp_xoffset_min": null,

  "pp_xoffset_max": null,

  "pp_yoffset_min": null,

  "pp_yoffset_max": null,

  "useragent": null,

  "br_name": null,

  "br_family": null,

  "br_version": null,

  "br_type": null,

  "br_renderengine": null,

  "br_lang": null,

  "br_features_pdf": true,

  "br_features_flash": false,

  "br_features_java": null,

  "br_features_director": null,

  "br_features_quicktime": null,

  "br_features_realplayer": null,

  "br_features_windowsmedia": null,

  "br_features_gears": null,

  "br_features_silverlight": null,

  "br_cookies": null,

  "br_colordepth": null,

  "br_viewwidth": null,

  "br_viewheight": null,

  "os_name": null,

  "os_family": null,

  "os_manufacturer": null,

  "os_timezone": null,

  "dvce_type": null,

  "dvce_ismobile": null,

  "dvce_screenwidth": null,

  "dvce_screenheight": null,

  "doc_charset": null,

  "doc_width": null,

  "doc_height": null,

  "tr_currency": null,

  "tr_total_base": null,

  "tr_tax_base": null,

  "tr_shipping_base": null,

  "ti_currency": null,

  "ti_price_base": null,

  "base_currency": null,

  "geo_timezone": null,

  "mkt_clickid": null,

  "mkt_network": null,

  "etl_tags": null,

  "dvce_sent_tstamp": null,

  "refr_domain_userid": null,

  "refr_dvce_tstamp": null,

  "contexts_com_snowplowanalytics_snowplow_ua_parser_context_1": [

    {

      "useragentFamily": "IE",

      "useragentMajor": "7",

      "useragentMinor": "0",

      "useragentPatch": null,

      "useragentVersion": "IE 7.0",

      "osFamily": "Windows XP",

      "osMajor": null,

      "osMinor": null,

      "osPatch": null,

      "osPatchMinor": null,

      "osVersion": "Windows XP",

      "deviceFamily": "Other"

    }

  ],

  "domain_sessionid": "2b15e5c8-d3b1-11e4-b9d6-1681e6b88ec1",

  "derived_tstamp": "2021-11-20T00:03:57.886Z",

  "event_vendor": "com.snowplowanalytics.snowplow",

  "event_name": "link_click",

  "event_format": "jsonschema",

  "event_version": "1-0-0",

  "event_fingerprint": "e3dbfa9cca0412c3d4052863cefb547f",

  "true_tstamp": "2021-11-20T00:03:57.886Z"

}'
```

---

# Configure Snowplow Client for GTM Server Side
> Configure IP forwarding, sp.js hosting, custom paths, common event mapping, and entity merging for the Snowplow Client in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/snowplow-client-configuration/

> **Tip:** The [GTM Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) has a `user_data` property. To populate this, you can attach a context Entity to your events of this schema: `iglu:com.google.tag-manager.server-side/user_data/jsonschema/1-0-0`, which can be found on [Iglu Central](https://github.com/snowplow/iglu-central/blob/853357452300b172ebc113d1d75d1997f595142a/schemas/com.google.tag-manager.server-side/user_data/jsonschema/1-0-0).

## Forward User IP address

As the container sits between the website user and the Snowplow collector (or other downstream destinations), the users IP will be unknown to the destination. By enabling this option, the users IP address will be included in the events sent to Tags.

By disabling this, you are able to use GTM SS as a proxy which can string user IP addresses from requests. Many tags also offer this functionality at the tag level.

## Populate GAv4 Client Properties

Enabled by default, this option will populate additional properties which the GAv4 requires, that is useful if you want to forward your Snowplow events to the GAv4 Tag.

## sp.js settings

This setting allows for your GTM SS Container to serve your `sp.js` JavaScript Tracker file. This allows you to have first party hosting of your tracker without the need to set up separate hosting or use a third party CDN.

It is recommended to rename `sp.js` if enabling this setting, as many adblockers will block requests to files named `sp.js`. A random string is the best option here.

![sp.js settings section with the Serve sp.js from container checkbox checked and a sp.js name field containing the example value 776b5b25.js.](/assets/images/spjssettings-905ab9716df666e83d799fa929e3bc40.png)

You can request _any_ version of the Snowplow JavaScript Tracker with this setting enabled. e.g. `https://{{gtm-ss-url}}/3.1.6/776b5b25.js` will load v3.1.6, or `https://{{gtm-ss-url}}/2.18.2/776b5b25.js` will load v2.18.2.

## Additional Options

### Custom POST Path

As many ad blockers will block the default `/com.snowplowanalytics.snowplow/tp2` POST path, it is recommended to change this and then update your trackers initialization to use this custom POST path.

### Claim GET Requests

The default Snowplow path for GET requests is `/i`, as this is so short there is a chance it could conflict with other Clients within your GTM SS Container. If you'd only like your Snowplow Client to listen for POST requests, you can disable this GET endpoint with this setting.

### Include Original `tp2` Event

If using this Client to receive Snowplow Tracker events and then forward to a Snowplow Collector with the Snowplow Tag, you should leave this option enabled as it will allow the Snowplow Tag to forward the original tracker event with no extra processing.

### Include Original Self Describing Event

By default, the self describing event will be "shredded" into a key using the schema name as the key, this is a "lossy" transformation, as the Minor and Patch parts of the jsonschema version will be dropped. This flag populates the original, lossless, Self Describing Event as `x-sp-self_describing_event`.

Let's assume we have a self-describing event following the schema `iglu:com.acme/foobar/jsonschema/1-0-0`. By default, the option to _Include Original Self Describing Event_ is disabled. So the Snowplow client by default will include it in the common event as:

```json
"x-sp-self_describing_event_com_acme_foobar_1": {

    "foo": "bar"

}
```

In case the option to _Include Original Self Describing Event_ is enabled, then the Snowplow client, if it finds the original event (see note below), will also include it in the common event, resulting in:

```json
"x-sp-self_describing_event_com_acme_foobar_1": {

    "foo": "bar"

},

"x-sp-self_describing_event": {

    "schema": "iglu:com.acme/foobar/jsonschema/1-0-0",

    "data": {

        "foo": "bar"

    }

}
```

> **Note:** This option only makes sense when using GTM in a [**Server Side Tag Manager (pre-pipeline)**](/docs/destinations/forwarding-events/google-tag-manager-server-side/#configuration-options) architecture, because it only makes a difference when the input is a _raw_ Snowplow event.
>
> In a [**Destinations Hub (post-pipeline)**](/docs/destinations/forwarding-events/google-tag-manager-server-side/#configuration-options) architecture, this option **does not apply**. Effectively, it’s always disabled, regardless of the setting. In the example above, this would mean that the data will contain `x-sp-self_describing_event_com_acme_foobar_1`, but not`x-sp-self_describing_event`.

### Include Original Contexts Array

By default, the contexts will be "shredded" into separate keys using the context name as the key, this is a "lossy" transformation, as the Minor and Patch parts of the jsonschema version will be dropped. If you would like to keep the original "lossless" contexts array (as `x-sp-contexts`), enable this option.

## Advanced common event options

![Advanced common event options section showing four collapsed subsections: client\_id (with Use default settings checked), user\_id (with Use default settings checked), Snowplow Entities mapping (with Merge selected Snowplow entities unchecked), and Snowplow Self-describing Event Mapping (with Merge selected Snowplow self-describing event data unchecked).](/assets/images/advanced_common_options-87a6c18646fd2ad02e1d8ca7d6e8a49d.png)

### `client_id`

#### Use default settings for client\_id mapping in common event

By default the Snowplow Client sets the `client_id` as follows: If the Snowplow event has the `client_session` context entity attached, its `userId` property is used. Else the `domain_userid` atomic property is used. Disabling this option reveals the following table that allows you to override the default behavior.

#### Specify client\_id

You can use this table to specify the rules to set the `client_id` of the common event. For consistency downstream it is suggested to specify properties that apply to all Snowplow events (atomic or through global context entities). The columns of this table are:

- **Priority**: Using this column you can set the priority (higher values mean higher priority) with which the Client will look into the Snowplow event to locate the value to set the `client_id`.
- **Property name or path**: This column refers to the common event, so you can define alternative Snowplow properties using the `x-sp-` prefix before the enriched property name or nested path (using dot notation). Example values: `x-sp-network_userid` or `x-sp-contexts_com_acme_user_1.0.anonymous_identifier`.

### `user_id`

#### Use default settings for user\_id mapping in common event

By default the Snowplow Client sets the `user_id` from the corresponding `user_id` property of the Snowplow event. Disabling this option reveals the following table that allows you to override the default behavior.

#### Specify user\_id

You can use this table to specify the rules to set the `user_id` of the common event, which will override the default Snowplow Client behavior. For consistency downstream it is suggested to specify properties that apply to all Snowplow events (atomic or through global context entities). The columns of this table are:

- **Priority**: Using this column you can set the priority (higher values mean higher priority) with which the Client will look into the Snowplow event to locate the value to set the `user_id`.
- **Property name or path**: This column refers to the common event, so you can define alternative Snowplow properties using the `x-sp-` prefix before the enriched property name or nested path (using dot notation). For example: `x-sp-contexts_com_acme_user_entity_1.0.email`.

### Snowplow Entities Mapping

> **Note:** This is an advanced feature and we recommend that you first explore the configuration options of your Tag(s) in case they are enough to cover your use case.

#### Merge selected Snowplow entities

Enable this option to allow merging of Snowplow context data to the Common Event. Checking this box reveals the following table:

![Snowplow Entities Mapping section with the Merge selected Snowplow entities checkbox checked, revealing a Context entities to merge table with columns for Schema, Apply to all versions, Prefix, Merge to, Custom path, Keep original mapping, and Custom transformation, with an Add Row button.](/assets/images/context_merging_overview-583c306bceab0a6601d86760ed9ebb73.png)

#### Context entities to merge

Using this table you can specify the rules to merge Snowplow context entity data to the Common Event. The columns of this table are:

![New row dialog for a context entity mapping rule, with fields for Schema (empty), Apply to all versions (False), Prefix (empty), Merge to (Event Properties), Custom path (empty), Keep original mapping (Keep), and Custom transformation (empty).](/assets/images/context_merging_new_row-95c55b188e68a938e0f393b772b1c3ab.png)

1. **Schema**: (Required) The schema of the context entity to merge.

> **Info:** The **Schema** can be specified in 3 ways:
>
> - Iglu URI (e.g. `iglu:com.acme/test/jsonschema/1-0-0` )
> - Enriched name (e.g. `contexts_com_acme_test_1`)
> - Common Event name (e.g. `x-sp-contexts_com_acme_test_1`)

2. **Apply to all versions**: (False/True) Whether the rule applies to all versions of the context entity schema. Default is False.

> **Info:** When you set **Apply to all versions** to `True`, it will apply the same rule to all versions of the schema, independently of the one mentioned in the **Schema** field.

3. **Prefix**: (Optional) Specify a prefix to use for property names when merging.

4. **Merge to**: (Event Properties/Custom) Specify where to merge the context entity's properties. Default is Event Properties, i.e. to the root level of the common event.

5. **Custom path**: (Optional) The path for custom merging.

> **Info:** The **Custom path** option applies only if the **Merge to** column is set to `Custom`, else the row is considered invalid.

6. **Keep original mapping**: (True/False) Whether to keep the original context mapping. Default is True.

7. **Custom transformation**: (Optional) Specify a Variable returning **a function** that represents a custom transformation of the context data to the desired object before merging. The default behaviour is the following: It first checks whether the context array contains a single entity object and if so, it merges this single object. In case the context specified contains multiples of entity objects, the rule is not applied (in those cases you will need to provide a custom transformation function through a Variable).

> **Info:** The function signature must be: `(contextArray, event) => Object`
>
> The Client guarantees that it will call this function providing as arguments the original context (Array) specified and the Common Event (Object) it has constructed. The event argument is provided in order to enable merging logic to optionally be based on event properties. Please note that it is not possible to modify the event inside the function you will define. The Client expects the function to return an Object, otherwise the value is ignored.

#### Example: Custom transformation function for context mapping

You can define and return the function in a **Variable Template**, which you can then use to create a **Variable** to reference.

The following is an example code of such a **Variable Template**:

```javascript
// Variable template code

function selectFirst(contextArray, event) {

  // the function must return an object

  return contextArray[0];

}



// The Variable must return the function

return selectFirst;
```

### Snowplow Self-describing Event Mapping

> **Note:** This is an advanced feature and we recommend that you first explore the configuration options of your Tag(s) in case they are enough to cover your use case.

#### Merge selected Snowplow self-describing event data

Enable this option to allow merging of Snowplow self-describing event data to the Common Event. Checking this box reveals the following table:

![Snowplow Self-describing Event Mapping section with the Merge selected Snowplow self-describing event data checkbox checked, revealing a Self-describing events to merge table with columns for Schema, Apply to all versions, Prefix, Merge to, Custom path, Keep original mapping, and Custom transformation, with an Add Row button.](/assets/images/selfdesc_merging_overview-b2be0321a3de3b7d0bdf95169cf9b5b7.png)

#### Self-describing events to merge

Using this table you can specify the rules to merge Snowplow self-describing event data to the Common Event. The columns of this table are:

![New row dialog for a self-describing event mapping rule, with fields for Schema (empty), Apply to all versions (False), Prefix (empty), Merge to (Event Properties), Custom path (empty), Keep original mapping (True), and Custom transformation (empty).](/assets/images/selfdesc_merging_new_row-f75e06e030456a7f31905c40e6b6de93.png)

**Schema**: (Required) The schema of the self-describing event.

> **Info:** The **Schema** can be specified in 3 ways:
>
> - Iglu URI (e.g. `iglu:com.acme/myevent/jsonschema/1-0-0` )
> - Enriched name (e.g. `contexts_com_acme_myevent_1`)
> - Common Event name (e.g. `x-sp-contexts_com_acme_myevent_1`)

**Apply to all versions**: (False/True) Whether the rule applies to all versions of the self-describing event schema. Default is False.

> **Info:** When you set **Apply to all versions** to `True`, it will apply the same rule to all versions of the schema, independently of the one mentioned in the **Schema** field.

**Prefix**: (Optional) Specify a prefix to use for property names when merging.

**Merge to**: (Event Properties/Custom) Specify where to merge the self-describing event’s properties. Default is Event Properties, i.e. to the root level of the common event.

**Custom path**: (Optional) The path for custom merging.

> **Info:** The **Custom path** option applies only if the **Merge to** column is set to `Custom`, else the row is considered invalid.

**Keep original mapping**: (True/False) Whether to keep the original context mapping. Default is True.

**Custom transformation**: (Optional) Specify a Variable returning **a function** that represents a custom transformation of the self-describing data to the desired object before merging. The default behaviour is to merge the self-describing data object as is.

> **Info:** The function signature must be: `(selfDescObject, event) => Object`
>
> The Client guarantees that it will call this function providing as arguments the original self-describing data (Object) specified and the Common Event (Object) it has constructed. The event argument is provided in order to enable merging logic to optionally be based on other event properties. Please note that it is not possible to modify the event inside the function you will define. The Client expects the function to return an Object, otherwise the value is ignored.

#### Example: Custom transformation function for self-describing event mapping

You can define and return the function in a **Variable Template**, which you can then use to create a **Variable** to reference.

The following is an example code of such a **Variable Template**:

```javascript
// Variable template code

const Object = require('Object');



function addKeySuffix(selfDescObject, event) {

  // the function must return an object

  return Object.keys(selfDescObject).reduce((acc, curr) => {

    acc[curr.concat('_suffix')] = selfDescObject[curr];

    return acc;

  }, {});

}



// The Variable must return the function

return addKeySuffix;
```

---

# Snowplow Tag for GTM Server Side
> Forward events to Snowplow Collector from GTM Server Side using the Snowplow Tag, supporting events from Snowplow JavaScript Tracker or other GTM SS clients like GA4.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/

The [Snowplow Tag for GTM SS](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-server-side-tag) is most useful if using GTM SS as a Server Side Tag Manager for Snowplow JavaScript Tracker events, as you will want to ensure you forward these events to your Snowplow Collector. The Snowplow Tag will automatically forward any events the Snowplow Client receives once it has been configured with your Collector URL.

The Snowplow Tag can also construct Snowplow events from other GTM SS Clients such as GAv4.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM SS, click "Search Gallery" in the Tag Templates section
2. Search for "Snowplow" and select the official "By Snowplow" tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking "Add"

## Snowplow Tag Setup

With the template installed, you can now add the Snowplow Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the Snowplow Tag as your Tag Configuration
2. Select your desired Trigger - If using the Snowplow JavaScript Tracker and Snowplow Client, you want to select "All Events"
3. Enter your Snowplow Collector Endpoint, and confirm the Cookie Name matches that of your Collector
4. Click Save

![Animated walkthrough of adding a Snowplow Tag in the GTM Server Side Tags tab, from an empty container through tag configuration and saving](/assets/images/tagsetup-3a2fbd7ede526d18786087b5b2b1dfbb.gif)

---

# Event settings for Snowplow Tag in GTM SS
> Configure structured events, self-describing events, and context entities for the Snowplow Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/snowplow-tag-configuration/advanced-event-settings/

This page describes the event settings available in the Snowplow Tag for GTM Server Side.

## Structured events

This section allows you to specify which incoming events will be tracked as custom Snowplow events. It only targets events that follow the event/category/label/value Universal Analytics model.

### Send selected events as Snowplow Structured Events

Enabling this check-box reveals a multi-line text field to allow you to set custom Structured Events.

### Event name(s) selected

Add the event names (in separate lines) to be tracked as custom structured Snowplow events.

## Self-describing events

This section allows you to define [Snowplow self-describing events](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-event-self-describing).

![The Self-describing events section with page\_interaction mapped to iglu.acme/page\_interaction/jsonschema/1-0-0, and event definitions mapping interactionType and label properties from client event properties](/assets/images/self_describing_setup-5b8b1628a6630ae2a9c8d69ea3959bb0.png)

### Define events to be sent as Snowplow Self-Describing Events

Enable this to allow custom self-describing event definitions.

### Event Name to Schema

A table of the events (event names and corresponding schemas) to be tracked as custom self-describing Snowplow events.

Add events which you would like to capture and convert into Self Describing Events. The `Event Name` should equal the Clients `event_name` property, if this is found when the Snowplow Tag fires, this tag will create a Self Describing Event using the specified schema.

### Event Definitions

A table of definitions for self-describing data properties. Each row maps a single data property of a custom self-describing Snowplow event to its value.

For each Event, you can also read properties off the client event object and add them as properties to the Self Describing Event.

## Context entities

### Apply context entities settings also to Snowplow events

Enable this tick box to also apply the Context entities settings to raw Snowplow events. This may be helpful in use cases where you want to modify the entities already attached to Snowplow events **before** they are relayed to your Snowplow collector.

### Context entities settings

This section allows you to define the context entities to be attached to your events using the following subsections.

You can define both custom and global context entities: Custom Entities are attached where the event\_name matches the incoming event, whereas Global Entities will be applied to all events.

Both subsections offer two ways to set your context entities:

1. Using the default tables in the UI:

![The context entities settings showing empty Custom context entities and Global context entities tables for defining entity mappings using the default table UI](/assets/images/context_entities_settings-30d068889d497c4542ba61cf9f44e4aa.png)

2. Through GTM Server-side variables (best suited for advanced use-cases):

![The context entities settings with both \&quot;Use variables\&quot; options enabled, revealing variable-based configuration fields for custom and global context entities](/assets/images/context_entities_settings_with_var-32d664fa3c5b3d2c3172337cd0b69b6f.png)

#### Custom context entities

> **Tip:** Custom context entities can be used to augment any standard Snowplow event, including self describing events, with additional data. The entities set in this subsection are attached _where the event name matches the incoming event_.

##### Use variables to define custom context entities

Enabling this setting allows you to set custom context entities using variables that return the entities array to attach to the event.

##### Define custom context entities

This is the way to define custom context entities through table UI. The columns of this table are:

- **Event Name**: The event that this entity will be attached to.

- **Entity Schema**: The schema of the entity.

- **Entity Property Name**: The key path to the entity's data.

- **Type**: The type of the value. Available options are:

  - **Default**: This option means that the value remains to its original type.
  - **String**: This option means that the value will be interpreted as a string.
  - **Boolean**: This option means that the value will be interpreted as a boolean.
  - **Number**: This option means that the value will be interpreted as a number.

- **Reference**: What the value references. Available options are:

  - **Client Event Property** (default): This means that what is set in the **Value** column corresponds to a property path in the client event.
  - **Constant or Variable**: This means that what is set in the **Value** column is the actual value to be used.

- **Value**: The value to set the property name to.

##### Set custom context entities through variables

This table is revealed if **Use variables to define custom context entities** option is enabled. It provides an alternative way to set custom context entities for an even through a GTM Server-side variable. The variable specified must return an array of context entities.

The columns of this table are:

- **Event Name**: The event that this entity will be attached to.
- **Custom Entities Array**: The variable containing the array of entities to be attached to the event.

#### Global context entities

> **Tip:** Global context entities are custom entities that apply globally. This lets you define your own context entities once and have them sent _with **all** events_.

##### Use a variable to define global context entities

Enabling this setting allows you to set global context entities using a variable that returns the entities array to attach to the event.

##### Define global context entities

This is the way to define global context entities through a table UI. The columns of this table are:

- **Entity Schema**: The schema of the entity.

- **Entity Property Name**: The key path to the entity's data.

- **Type**: The type of the value. Available options are:

  - **Default**: This option means that the value remains to its original type.
  - **String**: This option means that the value will be interpreted as a string.
  - **Boolean**: This option means that the value will be interpreted as a boolean.
  - **Number**: This option means that the value will be interpreted as a number.

- **Reference**: What the value references. Available options are:

  - **Client Event Property** (default): This means that what is set in the **Value** column corresponds to a property path in the client event.
  - **Constant or Variable**: This means that what is set in the **Value** column is the actual value to be used.

- **Value**: The value to set the property name to.

##### Set global context entities through variable

This text box is revealed if **Use a variable to define global context entities** option is enabled.

### Examples

In the following screenshots you can find examples of context entities settings using:

#### 1. The default table UI

_**Scenario**_: On a tutorial platform, we use GTM to send a `tutorial_begin` event type to our GTM Server-side container with parameters:

```javascript
dataLayer.push({

  'event': 'tutorial_begin',

  'tutorial.id': 'math101',

  'tutorial.category_name': 'mathematics',

  'user_data.email_address': 'foo@bar.baz',

});
```

We are using the Snowplow GTM SS Tag to forward it to our Snowplow pipeline, and we want to attach context entities. For our example, let's say:

- our `tutorial` Snowplow entity corresponds to this schema

  ```json
  {

      "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

      "self": {

          "vendor": "com.acme",

          "name": "tutorial",

          "version": "1-0-0",

          "format": "jsonschema"

      },

      "description": "Tutorial information",

      "type": "object",

      "properties": {

          "id": {

              "description": "The ID of the tutorial",

              "type": "string",

              "maxLength": 4096

          },

          "category": {

              "description": "The category of the tutorial",

              "type": ["string", "null"],

              "maxLength": 4096

          }

      },

      "required": ["id"]

      "additionalProperties": false

  }
  ```

- we also want user information, so we will use the `user_data` context entity ([available in Iglu Central](https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.tag-manager.server-side/user_data/jsonschema/1-0-0)) attached to all events as well.

Then we can configure the Snowplow GTM SS Tag's Context entities settings as:

![The context entities settings example with tutorial\_begin custom entities mapping id and category fields from client event properties, and a global user\_data entity mapping email\_address](/assets/images/context_entities_settings_example-b75aaf07d445a7e76d034b91aac9270c.png)

Here we have configured the Context entities settings in order to:

1. Attach the `tutorial` entity only to `tutorial_begin` event (using the **Custom context entities** section). Consequently we specify how to derive the data of the entity:

- We set the `id` to the value found in the `tutorial.id` key path of the client event
- We set the `category` property to the value found in the `tutorial.category_name` key path of the client event

2. Attach the `user_data` entity to all events (using the **Global context entities** section). In the row, we specify how to derive the `user_data` entity's data:

- We set the `email_address` to the value found in the `user_data.email_address` key path of the client event

> **Note:** As you can see in the images above, the schema for an entity can be written in 2 equally correct ways:
>
> 1. With the `iglu:` prefix, e.g. `iglu:com.acme/product/jsonschema/1-0-0`
> 2. Without the `iglu:` prefix, e.g. `com.acme/product/jsonschema/1-0-0`

#### 2. GTM SS Variables

As mentioned, it is also possible to reference GTM Server-side Variables in order to set both the Custom and the Global context entities. As a simple example:

![The context entities settings example using GTM variables, with {product\_entities} attached to purchase events and {global\_entities} applied to all events globally](/assets/images/context_entities_settings_example_var-a10e8e255f4ca4506f15accc03749520.png)

Here:

1. We specify the Variable `product_entities` as the context entities to attach only to `purchase` events
2. We specify the Valiable `global_entities` as the context entities to attach to all events

The Variables referenced must return an array of context entities. For example, the return value of such a Variable should look like:

```javascript
[

  {

    schema: "iglu:com.example_company/page/jsonschema/1-2-1",

    data: {

      pageType: 'test',

      lastUpdated: '2022-11-18T17:59:00',

    }

  },

  {

    schema: "iglu:com.example_company/user/jsonschema/2-0-0",

    data: {

      userType: 'tester',

    }

  }

]
```

## Additional event settings

### Base64 encoding

Whether to encode the custom self-describing event data in base64.

### Platform identifier

When a platform is not specified on the event, this value will be used.

### App ID

This text box allows you to specify the `app_id` for the event.

> **Note:** In case you are specifying the `app_id` through a GTM-SS Variable, please ensure that its return value is a string, otherwise it will be ignored.

---

# Configure Snowplow Tag for GTM Server Side
> Configure collector URL, cookie settings, advanced event settings, and logging for the Snowplow Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/snowplow-tag-configuration/

## Collector URL (Required)

Set this to the URL of your Snowplow Collector you wish to send events to.

## Cookies Settings

If you have configured your Snowplow collector to have different cookie details, you should ensure they match here.

### Name (Required)

This value must match the value of your Collector cookie name, this will allow the Snowplow Tag to find and return your Collector Cookie back to your users browser.

### Override Cookie Properties

If you'd like to overwrite your collectors cookie settings you can do that here.

#### Domain Override

To return a cookie with a different domain value than your collector, you can override it to another string here. "auto" will ensure the value is unchanged.

#### Path Override

This will override the path value of the cookie.

#### SameSite Override

This will override the SameSite flag on the cookie.

#### Expiration Override in Seconds

Allows the expiration time of the cookie to be altered. This value is in seconds, and defaults to 63072000 (2 years).

#### HttpOnly

Overwrites the HttpOnly flag on the cookie. Will be `true` if enabled (default), or `false` if disabled.

#### Secure

Overwrites the Secure flag on the cookie. Will be `true` if enabled (default), or `false` if disabled. Setting this to `false` and `SameSite` to `None` will prevent browsers from being able to store the cookie.

### Cookie Headers

![The Cookie Headers section with \&quot;Forward Cookies in the request headers\&quot; enabled, showing a cookie named \&quot;foo\&quot; with the Decode dropdown set to \&quot;No\&quot;](/assets/images/cookie_headers-acafd18e7933e0e939c4ff547256a52a.png)

#### Forward Cookies in the request headers

This option allows you to forward selected cookies in the request headers and it is disabled by default. Enabling it reveals the "Cookies to forward" table.

#### Cookies to forward

Using this table allows you to select the cookies to forward. Its columns are:

- **Cookie Name**: The name of the cookie to forward
- **Decode**: Whether to want to decode the cookie value(s) before forwarding (defaults to `No`)

## Advanced Event Settings

![The Advanced Event Settings section showing four collapsed subsections: Structured events, Self-describing events, Context entities, and Additional event settings](/assets/images/advanced_event_settings_overview-6567e2661a9a168f1271f37ac28335ae.png)

This section allows you to:

- Specify events to be tracked as Structured events
- Configure Self-describing event definitions
- Specify Context entities to be attached to events
- Customize additional event settings

You can find out more about all the available configuration options in the [Advanced Event Settings page](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/snowplow-tag-configuration/advanced-event-settings/).

## Logs Settings

Through the Logs Settings you can control the logging behavior of the Snowplow Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the default option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Snowplow GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{

    "Name": "Snowplow", // the name of the tag

    "Type": "Message",  // the type of log (one of "Message", "Request", "Response")

    "TraceId": "xxx",   // the "trace-id" header if exists

    "EventName": "xxx"  // the name of the event the tag fired at

}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | "Message"                                                      |
| Request     | "RequestMethod", "RequestUrl", "RequestHeaders", "RequestBody" |
| Response    | "ResponseStatusCode", "ResponseHeaders", "ResponseBody"        |

---

# Debug and test GTM Server Side tags
> Test and debug Google Tag Manager Server Side tag configurations using Preview Mode before deploying to production environments.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/testing/

If you are working on some changes to the configuration of your Google Tag Manager tags and would like to test them before applying them in production, you can use GTM’s [Preview Mode](https://developers.google.com/tag-platform/tag-manager/server-side/debug) feature. It shows information about the events it receives, which tags get triggered, etc.

You can direct some (or all) of your Snowplow events to the Preview Mode, instead of the production tags.

> **Note:** To follow the steps below, you will need to be running [Snowbridge](/docs/api-reference/snowbridge/) 2.3+. You will also need to have the [`spGtmssPreview` transformation](/docs/api-reference/snowbridge/configuration/transformations/builtin/spGtmssPreview/) activated (this is the default for Snowplow customers using Snowbridge with GTM Server Side).

## Copy the preview header

Once you enter Preview Mode in Google Tag Manager, click on the three dots in the top right corner of the screen and click “Send requests manually”.

You will see a popup with a preview header, for example (not a real value):

> **Note:**```text
> sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo=
> ```

Copy the header value (in the example above, `sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo=`).

## Add the preview header to your events

You can add the header value to all or some of your events as an [entity](/docs/fundamentals/entities/).

For example, if you are using the [JavaScript tracker](/docs/sources/web-trackers/):

```javascript
snowplow('trackPageView', {

  context: [{

    schema: 'iglu:com.google.tag-manager.server-side/preview_mode/jsonschema/1-0-0',

    data: {

      // paste your header value here

      'x-gtm-server-preview': 'sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo='

    }

  }]

});
```

You can also add it as a [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) for all events:

```javascript
const gtmPreviewContext = {

  schema: 'iglu:com.google.tag-manager.server-side/preview_mode/jsonschema/1-0-0',

  data: {

    // paste your header value here

    'x-gtm-server-preview': 'sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo='

  }

};

snowplow('addGlobalContexts', [gtmPreviewContext]);
```

For trackers other than the JavaScript tracker, the approach is the same — you need to add the preview header as an entity.

> **Tip:** Note that the header value can change over time as you make changes to your Google Tag Manager setup.

---

# Real-time event forwarding to third-party platforms
> Send Snowplow events to third-party platforms in real-time using Snowplow's managed event forwarding solution with built-in filtering, field mapping, and JavaScript transformations.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/

Event forwarders let you filter, transform, and send Snowplow events to third-party platforms in real-time. They're deployed as fully managed apps that sit alongside warehouse and lake loaders in your Snowplow cloud account. You can configure forwarders through Snowplow Console.

![Event forwarding architecture showing data flow from Snowplow pipeline through forwarders to destination APIs](/assets/images/event-forwarding-diagram-2e83ef2253fd7137b01a0f5a81e2aab4.svg)

Event forwarding uses [Snowbridge](/docs/api-reference/snowbridge/) under the hood, deployed within your existing Snowplow cloud account, to transform and deliver events reliably. For detailed setup guides and field mappings, check out the list of [available integrations](/docs/destinations/forwarding-events/integrations/). For complex requirements or unsupported destinations, [advanced alternatives](#alternative-approaches) are also available.

## Use cases

Event forwarding works best for use cases where you need low-latency event delivery and don't require complex aggregations across multiple events. For more complex transformations or batch processing, consider using [reverse ETL](/docs/destinations/reverse-etl/) instead.

Event forwarding is a good fit for use cases such as:

- **Real-time personalization**: send events to marketing automation or customer engagement platforms for immediate campaign triggers
- **Product analytics**: forward user actions to analytics tools for real-time product insights
- **A/B testing**: send experiment events to testing platforms for real-time optimization and analytics
- **Fraud detection**: forward security-relevant events to monitoring systems
- **Customer support**: stream events to support platforms for context-aware assistance

## How it works

Event forwarders are deployed as managed [Snowbridge](/docs/api-reference/snowbridge/) apps that consume events from your enriched event stream in near real-time. It uses a [JavaScript transformation function](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) generated from your configuration to filter and transform events.

Here's how forwarders process events:

1. **Read events**: reads enriched events from your stream (Kinesis, Pub/Sub, or EventHub) as the Snowplow pipeline produces them
2. **Apply filters**: checks each event against your configured [JavaScript filters](/docs/destinations/forwarding-events/reference/#event-filtering) to decide whether to forward it
3. **Transform data**: transforms matching events using [field mapping expressions](/docs/destinations/forwarding-events/reference/#field-mapping) and custom JavaScript to convert Snowplow event data into your destination's API format
4. **Delivery handling**: sends transformed events to the destination via HTTP API calls. Retries failures depending on the [failure type](/docs/destinations/forwarding-events/event-forwarding-monitoring-and-troubleshooting/#failure-types-and-handling) and [logs non-retryable failures](/docs/destinations/forwarding-events/event-forwarding-monitoring-and-troubleshooting/#what-happens-when-events-fail) to cloud storage

The end-to-end latency from event collection to destination delivery is on the order of seconds. Latency depends on overall pipeline event volume, complexity of transformation logic, and destination rate limits.

## Getting started

To set up a new event forwarder, you must first create a **connection**, which stores the credentials and endpoint details needed to send events to your destination, and then a **event forwarder** configuration, which defines which pipeline to read events from and the transformations to apply to your events. For a step-by-step guide, see [creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

Each destination has its own requirements for API credentials, configuration, and field mappings. See the [available integrations](/docs/destinations/forwarding-events/integrations/) for destination-specific guides.

For detailed information on supported JavaScript expressions, field transformations, and mapping syntax, see the [filter and mapping reference](/docs/destinations/forwarding-events/reference/).

## Alternative approaches

Using event forwarders is the recommended starting point for most real-time delivery use cases. For more complex requirements or unsupported destinations, consider these alternatives:

- **[Snowbridge](/docs/api-reference/snowbridge/)**: flexible event routing with custom transformations and destinations (Kafka, Kinesis, HTTP APIs). Use when you need destinations not yet supported by event forwarders, complex custom transformations, non-HTTP destinations, or advanced batching and retry configurations.
- **[Google Tag Manager Server Side](/docs/destinations/forwarding-events/google-tag-manager-server-side/)**: use GTM SS to relay enriched events to destinations using rich libraries of tags. Best if your organization is heavily invested in GTM or if you need destinations not yet supported by event forwarders, but supported by GTM SS, such as Google Analytics.
- **[Custom integrations](/docs/destinations/forwarding-events/custom-integrations/)**: build your own solutions using AWS Lambda, GCP Cloud Functions, or other stream processing systems for fully bespoke requirements.

---

# Forward events to Amplitude
> Send Snowplow events to Amplitude for product analytics and behavioral insights using the HTTP API v2 with support for event tracking and user properties.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/amplitude/

Send Snowplow events to Amplitude to power product and marketing analytics or guide and survey personalization using Amplitude's [HTTP API v2](https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/).

## Prerequisites

Before setting up the forwarder in Console, you'll need an Amplitude API Key. To find your API key:

1. Log in to your Amplitude workspace
2. Create an HTTP API source (**Data** > **Sources**)
3. Copy the **API Key** from your source's **Settings** section

> **Tip:** To avoid introducing bad data in your production Amplitude project, we recommend using a test or development Amplitude project to test your transformations first. Then, create a new Connection in Console with your production API key, and a new forwarder that imports the configuration from your development forwarder.

## Getting started

### Configure the destination

To create the connection and forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Amplitude** for the connection type, enter your API key, and select the **Server Location** where your Amplitude project is hosted.

### Validate the integration

You can confirm events are reaching Amplitude by checking the **Ingestion Debugger** page in your Amplitude account:

1. From the left navigation bar, click **Data**, then select **Sources** from the sidebar. You will see a list of sources
2. Select the **Ingestion Debugger** tab
3. Filter the graphs to show only events from the **HTTP API** to confirm data is flowing as expected from Snowplow

## Sending custom properties

You can send custom properties beyond the standard fields defined in the schema reference below. Amplitude supports three types of custom properties:

- **event\_properties**: custom data associated with specific events (e.g., `event_properties.plan_type`, `event_properties.feature_flag`)
- **user\_properties**: custom data tied to user profiles (e.g., `user_properties.subscription_tier`, `user_properties.account_age`)
- **group\_properties**: custom data tied to groups when `event_type` is `$groupidentify` (requires Amplitude Accounts add-on)

For property names containing spaces, use bracket notation (e.g., `event_properties["campaign source"]`).

See Amplitude's [HTTP API v2 documentation](https://amplitude.com/docs/apis/analytics/http-v2#) for details on supported data types, property operations, and object depth limits. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Schema reference

This section contains information on the fields you can send to Amplitude, including field names, data types, required fields, and default Snowplow mapping expressions.

| Field                                  | Details                                                                                                                                                                                                                                                                                        |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `event_type` _string_                  | _Required._ A unique identifier for your event.Default mapping: `event?.event_name`                                                                                                                                                                                                            |
| `user_id` _unknown_                    | _Optional._ ID for the user. Required if `device_id` isn't provided.Default mapping: `event?.user_id`                                                                                                                                                                                          |
| `device_id` _unknown_                  | _Optional._ A device-specific identifier, such as the Identifier for Vendor on iOS. Required if `user_id` isn't provided.Default mapping: `event?.domain_userid ?? event?.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0]?.userId`                                               |
| `event_id` _integer_                   | _Optional._ An incrementing counter to distinguish events with the same user\_id and timestamp from each other.                                                                                                                                                                                |
| `session_id` _integer_                 | _Optional._ The start time of the session in milliseconds since epoch (Unix timestamp). Necessary if you want to associate events with a particular system.Default mapping: `spTstampToEpochMillis(event?.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0]?.firstEventTimestamp)` |
| `insert_id` _string_                   | _Optional._ A unique identifier for the event. Amplitude deduplicates subsequent events sent with the same device\_id and insert\_id within the past 7 days.Default mapping: `event?.event_id`                                                                                                 |
| `time` _integer_                       | _Optional._ The timestamp of the event in milliseconds since epoch. If time isn't sent with the event, then it's set to the request upload time.Default mapping: `spTstampToEpochMillis(event?.derived_tstamp)`                                                                                |
| `event_properties` _object_            | _Optional._ Arbitrary key-value pairs assigned to the event.                                                                                                                                                                                                                                   |
| `user_properties` _object_             | _Optional._ Arbitrary key-value pairs assigned to the user.                                                                                                                                                                                                                                    |
| `groups` _object_                      | _Optional._ Arbitrary key-value pairs representing groups of users.                                                                                                                                                                                                                            |
| `group_properties` _object_            | _Optional._ Arbitrary key-value pairs assigned to the groups listed in the `groups`.                                                                                                                                                                                                           |
| `$skip_user_properties_sync` _boolean_ | _Optional._ When true user properties aren't synced. Defaults to false.Default mapping: `false`                                                                                                                                                                                                |
| `app_version` _string_                 | _Optional._ The current version of your application.Default mapping: `event?.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.version`                                                                                                                                                |
| `platform` _string_                    | _Optional._ Platform of the device.Default mapping: `event?.platform`                                                                                                                                                                                                                          |
| `os_name` _string_                     | _Optional._ The name of the mobile operating system or browser that the user is using.Default mapping: `event?.os_name`                                                                                                                                                                        |
| `os_version` _string_                  | _Optional._ The version of the mobile operating system or browser the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.operatingSystemVersion`                                                                                                                  |
| `device_brand` _string_                | _Optional._ The device brand that the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceBrand`                                                                                                                                                             |
| `device_manufacturer` _string_         | _Optional._ The device manufacturer that the user is using.                                                                                                                                                                                                                                    |
| `device_model` _string_                | _Optional._ The device model that the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName`                                                                                                                                                              |
| `carrier` _string_                     | _Optional._ The carrier that the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.carrier`                                                                                                                                                                      |
| `country` _string_                     | _Optional._ The current country of the user.Default mapping: `event?.geo_country`                                                                                                                                                                                                              |
| `region` _string_                      | _Optional._ The current region of the user.Default mapping: `event?.geo_region`                                                                                                                                                                                                                |
| `city` _string_                        | _Optional._ The current city of the user.Default mapping: `event?.geo_city`                                                                                                                                                                                                                    |
| `dma` _string_                         | _Optional._ The current Designated Market Area of the user.                                                                                                                                                                                                                                    |
| `language` _string_                    | _Optional._ The language set by the user.Default mapping: `event?.br_lang`                                                                                                                                                                                                                     |
| `price` _number_                       | _Optional._ The price of the item purchased. Required for revenue data if the revenue field isn't sent. You can use negative values for refunds                                                                                                                                                |
| `quantity` _integer_                   | _Optional._ The quantity of the item purchased.                                                                                                                                                                                                                                                |
| `revenue` _number_                     | _Optional._ Revenue = (price x quantity). If you send all 3 fields of price, quantity, and revenue, then the revenue value is (price x quantity). Use negative values for refunds.                                                                                                             |
| `productId` _string_                   | _Optional._ An identifier for the item purchased. You must send a price and quantity or revenue with this field.                                                                                                                                                                               |
| `revenueType` _string_                 | _Optional._ The type of revenue for the item purchased. You must send a price and quantity or revenue with this field.                                                                                                                                                                         |
| `location_lat` _number_                | _Optional._ The current Latitude of the user.Default mapping: `event?.geo_latitude`                                                                                                                                                                                                            |
| `location_lng` _number_                | _Optional._ The current Longitude of the user.Default mapping: `event?.geo_longitude`                                                                                                                                                                                                          |
| `ip` _string_                          | _Optional._ The IP address of the user.Default mapping: `event?.user_ipaddress`                                                                                                                                                                                                                |
| `idfa` _string_                        | _Optional._ (iOS) Identifier for Advertiser.Default mapping: `event?.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.appleIdfa`                                                                                                                                                 |
| `idfv` _string_                        | _Optional._ (iOS) Identifier for Vendor.Default mapping: `event?.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.appleIdfv`                                                                                                                                                     |
| `adid` _string_                        | _Optional._ (Android) Google Play Services advertising ID.Default mapping: `event?.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.androidIdf`                                                                                                                                  |
| `android_id` _string_                  | _Optional._ (Android) Android ID (not the advertising ID).                                                                                                                                                                                                                                     |
| `plan` _object_                        | _Optional._ Tracking plan properties.Properties:* `branch` (string, optional): The tracking plan branch name.
* `source` (string, optional): The tracking plan source.
* `version` (string, optional): The tracking plan version.                                                              |

---

# Forward events to Bloomreach
> Send Snowplow events to Bloomreach Engagement for real-time customer analytics and personalization using the Batch Commands API.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/bloomreach/

Send Snowplow events to Bloomreach Engagement for real-time customer analytics, segmentation, and personalization using the [Batch Commands API](https://documentation.bloomreach.com/engagement/reference/batch-commands-2).

## Prerequisites

Before setting up the forwarder in Console, you'll need to configure an API key with 'Get' and 'Set' permission on all of the properties events that you intend to send to bloomreach. We recommend also permitting 'Get' and 'Set' permission on 'New Properties' and 'New Events', do avoid problems when new tracking is added.

You can configure this in the Access Management section of Project settings.

You'll also need the following from your Bloomreach Engagement account:

- **API Endpoint Host**: the base URL for your Bloomreach API instance (e.g., `api.exponea.com`)
- **Project Token**: a unique identifier for your Bloomreach project
- **API Key ID**: the identifier for your API key
- **API Secret**: the secret associated with your API key. Save the secret in a secure location, as you can only view it once after creation.

You can find all of these values in Bloomreach under **Project settings** > **Access management** > **API**.

> **Tip:** To avoid introducing test data in your production Bloomreach project, we recommend using a test or development project to test your transformations first. Then, create a new connection in Console with your production credentials, and a new forwarder that imports the configuration from your development forwarder.

## Getting started

### Configure the destination

To create the connection and forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Bloomreach** for the connection type and enter your API endpoint host, project token, API key ID, and API secret.

### Validate the integration

You can confirm events are reaching Bloomreach by checking the customer profiles in your Bloomreach Engagement project:

1. In Bloomreach, navigate to **Data & Assets** > **Customers**
2. Search for a customer whose events you have forwarded
3. Open the customer profile and check the **Events** tab to verify that your Snowplow events appear

## Identity management

Bloomreach uses a combination of hard and soft customer IDs to identify and merge customer profiles. The Snowplow integration defaults to using `user_id` as the hard ID (`registered`) and `domain_userid` as the soft ID (`cookie`).

When a user is anonymous, Bloomreach tracks their activity using the soft ID. When the user logs in and your event contains a `user_id` value, Bloomreach merges the anonymous profile with the identified profile, linking prior anonymous activity to the known customer.

The ID type names (`registered`, `cookie`) must match the customer ID types configured in your Bloomreach project. You can customize these mappings in the forwarder configuration if your project uses different ID type names.

## Sending custom properties

You can send custom event properties beyond the standard fields defined in the schema reference below. Custom properties are nested under the `properties` object. When configuring your forwarder, add field mappings formatted as `properties.your_custom_field` (e.g., `properties.plan_type`, `properties.feature_flag`).

For property names containing spaces, use bracket notation (e.g., `properties["referred by"]`).

See Bloomreach's [Batch Commands API documentation](https://documentation.bloomreach.com/engagement/reference/batch-commands-2) for details on supported data types and property requirements. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Schema reference

This section contains information on the fields you can send to Bloomreach, including field names, data types, required fields, and default Snowplow mapping expressions.

| Field                   | Details                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `customer_ids` _object_ | _Required._ Object identifying the customer. Must contain at least one ID. Keys are ID type names configured in the Bloomreach project.Properties:* `registered` (string, optional): Hard ID — the identified user ID. Used for cross-device identity resolution. Default mapping: `event.user_id`
* `cookie` (string, optional): Soft ID — anonymous visitor identifier, typically a browser cookie. Default mapping: `event.domain_userid`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `event_type` _string_   | _Required._ The name of the event type in Bloomreach.Default mapping: `event.event_name`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `timestamp` _integer_   | _Optional._ When the event occurred, as UNIX epoch seconds. Defaults to current time if omitted.Default mapping: `Math.floor(spTstampToJSDate(event.derived_tstamp).getTime() / 1000)`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `properties` _object_   | _Optional._ Arbitrary key-value pairs for the event. Values can be strings, numbers, booleans, or nested objects.Properties:* `url` (string, optional): The full URL of the page where the event occurred. Default mapping: `event.page_url`
* `title` (string, optional): The page title. Default mapping: `event.page_title`
* `referrer` (string, optional): The referring URL. Default mapping: `event.page_referrer`
* `ip` (string, optional): The user's IP address. Default mapping: `event.user_ipaddress`
* `city` (string, optional): The city of the user. Default mapping: `event.geo_city`
* `region` (string, optional): The region (state or province) of the user. Default mapping: `event.geo_region_name \|\| event.geo_region`
* `country` (string, optional): The country code of the user. Default mapping: `event.geo_country`
* `latitude` (number, optional): Latitude of the user's location. Default mapping: `event.geo_latitude`
* `longitude` (number, optional): Longitude of the user's location. Default mapping: `event.geo_longitude`
* `browser` (string, optional): Name of the browser. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.agentName`
* `browser_version` (string, optional): Version of the browser. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.agentVersion`
* `os` (string, optional): Operating system name. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.operatingSystemName \|\| event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.osType`
* `os_version` (string, optional): Operating system version. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.operatingSystemVersion \|\| event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.osVersion`
* `device` (string, optional): Device name or model. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName \|\| event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.deviceModel`
* `device_type` (string, optional): The type/class of device (e.g. Desktop, Mobile, Tablet). Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceClass`
* `device_manufacturer` (string, optional): Manufacturer of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceBrand \|\| event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.deviceManufacturer`
* `screen_height` (integer, optional): Device screen height in pixels. Default mapping: `event.dvce_screenheight`
* `screen_width` (integer, optional): Device screen width in pixels. Default mapping: `event.dvce_screenwidth`
* `language` (string, optional): The user's language/locale. Default mapping: `event.contexts_com_snowplowanalytics_snowplow_browser_context_1?.[0]?.browserLanguage \|\| event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.language`
* `timezone` (string, optional): The user's timezone. Default mapping: `event.os_timezone`
* `platform` (string, optional): The platform the event was tracked on (web, mob, etc.). Default mapping: `event.platform`
* `tracker` (string, optional): Tracker library that sent the event. Default mapping: `'Snowplow: ' + event.v_tracker`
* `utm_source` (string, optional): UTM source parameter. Default mapping: `event.mkt_source`
* `utm_medium` (string, optional): UTM medium parameter. Default mapping: `event.mkt_medium`
* `utm_campaign` (string, optional): UTM campaign parameter. Default mapping: `event.mkt_campaign`
* `utm_content` (string, optional): UTM content parameter. Default mapping: `event.mkt_content`
* `utm_term` (string, optional): UTM term parameter. Default mapping: `event.mkt_term`
* `carrier` (string, optional): Mobile carrier. Default mapping: `event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.carrier`
* `network_type` (string, optional): Network connection type (wifi, mobile, etc.). Default mapping: `event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.networkType`
* `app_version` (string, optional): Application version. Default mapping: `event.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.version \|\| event.contexts_com_snowplowanalytics_snowplow_application_1?.[0]?.version` |

---

# Forward events to Braze
> Send Snowplow events to Braze for real-time personalization and campaign automation using the Track Users API with support for user attributes, custom events, and purchases.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/braze/

Send Snowplow events to Braze for real-time personalization, user tracking, and campaign automation using Braze's [Track Users API](https://www.braze.com/docs/api/endpoints/user_data/post_user_track).

Snowplow supports the following Braze object types:

- **[User attributes](https://www.braze.com/docs/api/objects_filters/user_attributes_object)**: Profile data and custom user properties
- **[Custom events](https://www.braze.com/docs/api/objects_filters/event_object)**: User actions and behaviors
- **[Purchases](https://www.braze.com/docs/api/objects_filters/purchase_object)**: Transaction data with product details

## Prerequisites

Before setting up the forwarder in Console, you'll need the following from your Braze account:

- Braze REST API key with these permissions:

  - `users.track`
  - `users.alias.new`
  - `users.identify`
  - `users.export.ids`
  - `users.merge`
  - `users.external_ids.rename`
  - `users.alias.update`

- Braze REST API endpoint, found in Braze under **Settings** > **APIs and Identifiers**

## Getting started

### Configure the destination

To create the forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Braze** for the connection type and enter your API key and endpoint.

When configuring the forwarder, you can choose from the following **Braze object types** to map:

- **[Attributes](https://www.braze.com/docs/api/objects_filters/user_attributes_object)**: update user profile data
- **[Events](https://www.braze.com/docs/api/objects_filters/event_object)**: send custom user actions
- **[Purchases](https://www.braze.com/docs/api/objects_filters/purchase_object)**: send transaction events

### Validate the integration

You can confirm events are reaching Braze by checking the following pages in your Braze account:

1. Query Builder: in Braze, navigate to **Analytics** > **Query Builder**. You can write queries on the following tables to preview the data forwarded from Snowplow: `USER_BEHAVIORS_CUSTOMEVENT_SHARED`, `USERS_BEHAVIORS_PURCHASE_SHARED`.
2. API Usage Dashboard: in Braze, navigate to **Settings** > **API and Identifiers** to see a chart of API usage over time. You can filter specifically for the API key used by Snowplow and see both successes and failures.

## Sending custom properties

You can send custom properties beyond the standard fields defined in the schema reference below. The structure depends on which Braze object type you're using:

- **User attributes**: add as top-level fields (e.g., `subscription_tier`, `loyalty_points`)
- **Event properties**: nest under `properties` object (e.g., `properties.plan_type`, `properties.feature_flag`)
- **Purchase properties**: nest under `properties` object (e.g., `properties.color`, `properties.size`)

For property names containing spaces, use bracket notation (e.g., `["account type"]` or `properties["campaign source"]`).

See Braze's [Event Object documentation](https://www.braze.com/docs/api/objects_filters/event_object) for details on supported data types, property naming requirements, and payload size limits. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Schema reference

The sections below contain information on the fields you can send to Braze, including field names, data types, required fields, and default Snowplow mapping expressions.

### User attributes

Use the user attributes object to update standard user profile fields or add your own custom attribute data to the user.

| Field                                     | Details                                                                                                                                                                                                                          |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_update_existing_only` _boolean_         | _Optional._ If `true`, API requests will only update existing user profiles in Braze. `false` recommended.Default mapping: `false`                                                                                               |
| `user_alias` _object_                     | _Optional._ Braze user alias object.Properties:* `alias_name` (string, required): The actual value of the alias identifier.
* `alias_label` (string, required): Indicates the type of alias. E.g. `domain_userid`.               |
| `external_id` _string_                    | _Optional._ A unique user identifier. Required if `user alias`, `braze_id`, `email`, or `phone` is not provided.Default mapping: `event?.user_id`                                                                                |
| `braze_id` _string_                       | _Optional._ Identifier reserved for the Braze SDK. Required if `external_id`, `user alias`, `email`, or `phone` is not provided. Not recommended for use with the Snowplow integration.                                          |
| `country` _string_                        | _Optional._ ISO-3166-1 alpha-2 standard standard country code. Where the value does not meet that standard, Braze attempts to map it to a country. Where it cannot, the value will be NULL.Default mapping: `event?.geo_country` |
| `current_location` _object_               | _Optional._Properties:* `longitude` (number, optional): Longitude of the user's location.
* `latitude` (number, optional): Latitude of the user's location.                                                                      |
| `date_of_first_session` _string_          | _Optional._ Date at which the user first used the app. Must be ISO 8601 format.                                                                                                                                                  |
| `date_of_last_session` _string_           | _Optional._ Date at which the user most recently used the app. Must be ISO 8601 format.                                                                                                                                          |
| `dob` _string_                            | _Optional._ The user's date of birth.                                                                                                                                                                                            |
| `email` _string_                          | _Optional._ The user's email address.                                                                                                                                                                                            |
| `email_subscribe` _string_                | _Optional._ The user's email subscription status. Must be one of: `opted_in`, `unsubscribed`, `subscribed`                                                                                                                       |
| `email_open_tracking_disabled` _boolean_  | _Optional._ Set to true to disable the open tracking pixel from being added to all future emails sent to this user.                                                                                                              |
| `email_click_tracking_disabled` _boolean_ | _Optional._ Set to true to disable the click tracking for all links within future emails sent to this user.                                                                                                                      |
| `first_name` _string_                     | _Optional._ User's first name.                                                                                                                                                                                                   |
| `gender` _string_                         | _Optional._ The user's gender. Must be one of: `M`, `F`, `O`, `N`, `P`, `null`                                                                                                                                                   |
| `home_city` _string_                      | _Optional._ The user's city.                                                                                                                                                                                                     |
| `language` _string_                       | _Optional._ The user's preferred language. Must be an ISO-639-1 standard language code.Default mapping: `event?.br_lang`                                                                                                         |
| `last_name` _string_                      | _Optional._ User's last name.                                                                                                                                                                                                    |
| `marked_email_as_spam_at` _string_        | _Optional._ Date at which the user’s email was marked as spam. Must be ISO 8601 format.                                                                                                                                          |
| `phone` _string_                          | _Optional._ The user's phone number.                                                                                                                                                                                             |
| `push_subscribe` _string_                 | _Optional._ The user's push message subscription status. Must be one of: `opted_in`, `unsubscribed`, `subscribed`                                                                                                                |
| `push_tokens` _array of object_           | _Optional._ Array of objects with app\_id and token string.                                                                                                                                                                      |
| `time_zone` _string_                      | _Optional._ The user's time zone. Must be a valid IANA Time Zone.Default mapping: `event?.geo_timezone`                                                                                                                          |

### Events

Each event object represents a single occurrence of a custom event by a particular user at the specific time. You can set and use custom event properties in messages, data collection, and personalization.

| Field                             | Details                                                                                                                                                                                                                                |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `time` _string_                   | _Required._ Time of the event, required. Must be ISO 8601 format.Default mapping: `spTstampToJSDate(event?.collector_tstamp)?.toISOString()`                                                                                           |
| `name` _string_                   | _Required._ Name of the type of event.Default mapping: `event?.event_name`                                                                                                                                                             |
| `external_id` _string_            | _Optional._ A unique user identifier. Required if `user alias`, `braze_id`, `email`, or `phone` is not provided.Default mapping: `event?.user_id`                                                                                      |
| `braze_id` _string_               | _Optional._ Identifier reserved for the Braze SDK. Required if `external_id`, `user alias`, `email`, or `phone` is not provided. Not recommended for use with the Snowplow integration.                                                |
| `phone` _string_                  | _Optional._ The user's phone number.                                                                                                                                                                                                   |
| `user_alias` _object_             | _Optional._ Braze user alias object.Properties:* `alias_name` (string, required): The actual value of the alias identifier.
* `alias_label` (string, required): Indicates the type of alias. E.g. `domain_userid`.                     |
| `_update_existing_only` _boolean_ | _Optional._ If `true`, API requests will only update existing user profiles in Braze. `false` recommended.Default mapping: `false`                                                                                                     |
| `email` _string_                  | _Optional._ The user's email address.                                                                                                                                                                                                  |
| `app_id` _string_                 | _Optional._ Associates activity with a specific app in your Braze workspace. If set, should match a Braze App Identifier, found in Braze console's API section. Can be omitted, but incorrect values may result in data loss in Braze. |
| `properties` _object_             | _Optional._ Arbitrary key-value pairs assigned to the event in Braze.                                                                                                                                                                  |

### Purchases

The purchase object represents a user purchasing a single item by a user at a particular time. Each purchase object is located within a purchase array, which can represent a transaction with multiple items. The purchase object has fields that allow the Braze back-end to store and use this information for messages, data collection, and personalization.

| Field                             | Details                                                                                                                                                                                                                                |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `time` _string_                   | _Required._ Time of the purchase, required. Must be ISO 8601 format.                                                                                                                                                                   |
| `product_id` _string_             | _Required._ Identifier for the product purchased.                                                                                                                                                                                      |
| `currency` _string_               | _Required._ ISO 4217 Alphabetic Currency Code.                                                                                                                                                                                         |
| `price` _number_                  | _Required._ Price per item.                                                                                                                                                                                                            |
| `external_id` _string_            | _Optional._ A unique user identifier. Required if `user alias`, `braze_id`, `email`, or `phone` is not provided.                                                                                                                       |
| `braze_id` _string_               | _Optional._ Identifier reserved for the Braze SDK. Required if `external_id`, `user alias`, `email`, or `phone` is not provided. Not recommended for use with the Snowplow integration.                                                |
| `phone` _string_                  | _Optional._ The user's phone number.                                                                                                                                                                                                   |
| `user_alias` _object_             | _Optional._ Braze user alias object.Properties:* `alias_name` (string, required): The actual value of the alias identifier.
* `alias_label` (string, required): Indicates the type of alias. E.g. `domain_userid`.                     |
| `_update_existing_only` _boolean_ | _Optional._ If `true`, API requests will only update existing user profiles in Braze. `false` recommended.                                                                                                                             |
| `quantity` _integer_              | _Optional._ Quantity of the item purchased. Braze treats this as multiple individual purchases.                                                                                                                                        |
| `email` _string_                  | _Optional._ The user's email address.                                                                                                                                                                                                  |
| `app_id` _string_                 | _Optional._ Associates activity with a specific app in your Braze workspace. If set, should match a Braze App Identifier, found in Braze console's API section. Can be omitted, but incorrect values may result in data loss in Braze. |
| `properties` _object_             | _Optional._ Arbitrary key-value pairs assigned to the purchase in Braze.                                                                                                                                                               |

---

# Pre-built event forwarding integrations
> Pre-built event forwarding integrations for third-party destinations with authentication, field mapping, and API-specific configurations included.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/

Event forwarding supports third-party destinations through pre-built integrations that handle authentication, field mapping, and API-specific requirements.

## Available destinations

Snowplow event forwarding supports the following destinations:

- [Amplitude](/docs/destinations/forwarding-events/integrations/amplitude/)
- [Bloomreach](/docs/destinations/forwarding-events/integrations/bloomreach/)
- [Braze](/docs/destinations/forwarding-events/integrations/braze/)
- [Mixpanel](/docs/destinations/forwarding-events/integrations/mixpanel/)

---

# Forward events to Mixpanel
> Send Snowplow events to Mixpanel for product analytics and user behavior insights using the Import API with support for event tracking and custom properties.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/mixpanel/

Send Snowplow events to Mixpanel to power product analytics, user behavior tracking, and funnel analysis using Mixpanel's [Import API](https://developer.mixpanel.com/reference/import-events).

## Prerequisites

Before setting up the forwarder in Console, you'll need the following from your Mixpanel account:

- **Project ID**: found in Mixpanel under **Settings** > **Project Settings**
- **Service Account Username**: create a service account in Mixpanel under **Settings** > **Organization Settings** > **Service Accounts**. The service account must have either **Admin** or **Owner** permissions.
- **Service Account Password**: generated when you create the service account

> **Tip:** To avoid introducing test data in your production Mixpanel project, we recommend using a test or development Mixpanel project to test your transformations first. Then, create a new Connection in Console with your production credentials, and a new forwarder that imports the configuration from your development forwarder.

## Getting started

### Configure the destination

To create the connection and forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Mixpanel** for the connection type, enter your project ID, service account username, and service account password. You'll also need to select the **Server Location** where your Mixpanel project is hosted:

- **United States**: `https://api.mixpanel.com/`
- **European Union**: `https://api-eu.mixpanel.com/`
- **India**: `https://api-in.mixpanel.com/`

### Validate the integration

You can confirm events are reaching Mixpanel by checking the **Events** page in your Mixpanel account:

1. From the left navigation bar, click **Events**
2. You should see your Snowplow events appearing in the live view
3. You can also navigate to **Settings** > **Project Settings** > **Data & Exports** to view import statistics

## Identity management

Mixpanel uses a combination of `distinct_id` (user identifier) and `$device_id` (device identifier) to track users across sessions. The Snowplow integration defaults to using `user_id` for `distinct_id` and a coalesce of `domain_userid` and `client_sesion.user_id` for `$device_id`, which supports Mixpanel's [Simplified ID Merge system](https://docs.mixpanel.com/docs/tracking-methods/id-management#simplified-id-merge-system).

When a user logs in and your event contains a `user_id` value, Mixpanel will automatically merge the user's anonymous activity (tracked via `$device_id`) with their identified profile (tracked via `distinct_id`).

## Sending custom properties

You can send custom event properties beyond the standard fields defined in the schema reference below. Custom properties are nested under the `properties` object. When configuring your forwarder, add field mappings formatted as `properties.your_custom_field` (e.g., `properties.plan_type`, `properties.feature_flag`).

For property names containing spaces, use bracket notation (e.g., `properties["referred by"]`).

See Mixpanel's [Import Events API documentation](https://developer.mixpanel.com/reference/import-events) for details on supported data types and property requirements. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Schema reference

This section contains information on the fields you can send to Mixpanel, including field names, data types, required fields, and default Snowplow mapping expressions.

| Field                 | Details                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `event` _string_      | _Required._ The name of the eventDefault mapping: `event.event_name`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `properties` _object_ | _Required._Properties:* `time` (integer, required): The time at which the event occurred, in milliseconds since epoch. Default mapping: `spTstampToEpochMillis(event.derived_tstamp)`
* `distinct_id` (string, required): Identifies the user who performed the event. Default mapping: `event.user_id`
* `$insert_id` (string, required): A unique identifier for the event, used for deduplication. Default mapping: `event.event_id`
* `ip` (string, optional): The IP address of the user. Default mapping: `event.user_ipaddress`
* `$city` (string, optional): The city of the event sender. Default mapping: `event.geo_city`
* `$region` (string, optional): The region (state or province) of the event sender. Default mapping: `event.geo_region`
* `mp_country_code` (string, optional): The country of the event sender. Default mapping: `event.geo_country`
* `$browser` (string, optional): Name of the browser. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.agentName`
* `$browser_version` (string, optional): Version of the browser. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.agentVersion`
* `$current_url` (string, optional): The full URL of the webpage on which the event is triggered. Default mapping: `event.page_url`
* `$referrer` (string, optional): Referring URL including your own domain. Default mapping: `event.page_referrer`
* `$referring_domain` (string, optional): Referring domain including your own domain. Default mapping: `event.refr_urlhost`
* `$device` (string, optional): Name of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName`
* `$device_id` (string, optional): A unique device identifier. Used in Mixpanel's Simplified ID Merge API. Default mapping: `event.domain_userid ?? event.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0]?.userId`
* `$screen_height` (integer, optional): The height of the device screen in pixels. Default mapping: `event.dvce_screenheight`
* `$screen_width` (integer, optional): The width of the device screen in pixels. Default mapping: `event.dvce_screenwidth`
* `$os` (string, optional): Operating system of the device. Default mapping: `event.os_name`
* `$os_version` (string, optional): Operating system version. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.operatingSystemVersion`
* `$manufacturer` (string, optional): Manufacturer of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceBrand`
* `$model` (string, optional): Model of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName`
* `$app_version_string` (string, optional): The app version. Default mapping: `event.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.version`
* `$app_build_number` (string, optional): Build number for the mobile app. Default mapping: `event.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.build`
* `$carrier` (string, optional): Wireless carrier of the device owner. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.carrier`
* `$radio` (string, optional): The current cellular network communication standard (3G, 4G, LTE, etc). Default mapping: `event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.networkTechnology`
* `$wifi` (boolean, optional): Set to true if the user's device is connected to wifi. Default mapping: `event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.networkType === 'wifi'`
* `$user_id` (string, optional): The identified ID of the user. Used in Mixpanel's Simplified ID Merge API. Default mapping: `event.user_id`
* `$lib_version` (string, optional): Tracker library version. Default mapping: `event.v_tracker`
* `utm_source` (string, optional): UTM source parameter from the URL. Default mapping: `event.mkt_source`
* `utm_medium` (string, optional): UTM medium parameter from the URL. Default mapping: `event.mkt_medium`
* `utm_campaign` (string, optional): UTM campaign parameter from the URL. Default mapping: `event.mkt_campaign`
* `utm_content` (string, optional): UTM content parameter from the URL. Default mapping: `event.mkt_content`
* `utm_term` (string, optional): UTM term parameter from the URL. Default mapping: `event.mkt_term`
* `mp_lib` (string, optional): Tracker library that sent the event. Default mapping: `'Snowplow: ' + event.v_tracker`
* `mp_sent_by_lib_version` (string, optional): Mixpanel library version used to send data (not necessarily the same as the version which enqueued the data).
* `$screen_dpi` (integer, optional): Pixel density of the device screen.
* `$initial_referrer` (string, optional): Referring URL when the user first arrived on your site. Defaults to “$direct” if the user is not referred.
* `$initial_referring_domain` (string, optional): Referring domain at first arrival. Defaults to “$direct” if the user is not referred.
* `$search_engine` (string, optional): The search engine that the customer used when they arrived at your domain.
* `mp_keyword` (string, optional): Search keywords detected on the referrer from a search engine to your domain. This property is only collected when search keywords are included in a URL.
* `$watch_model` (string, optional): The model of the iOS watch.
* `$bluetooth_enabled` (boolean, optional): Set to true if Bluetooth is enabled, false if not.
* `$bluetooth_version` (string, optional): Set to “none”, “ble”, or “classic”.
* `$has_nfc` (boolean, optional): The device supports Near Field Communication(NFC). Set to true if the device hardware supports NFC, false if not.
* `$has_telephone` (boolean, optional): Set to true if this device has telephone functionality, false if not. |

---

# Manage event forwarders in Console
> Edit, clone, and delete Snowplow event forwarders in Console to update configurations, duplicate setups, or remove unused destinations.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/managing-forwarders/

This page explains how to edit, clone, and delete event forwarders. To start, go to **Destinations** > **Event forwarding** in [Snowplow Console](https://console.snowplowanalytics.com).

## Edit a forwarder

To edit a forwarder:

1. Click **Details** under the destination you want to change to open the destination details page.
2. On the event forwarders overview table, click the three dots next to the forwarder you want to change and select **Edit**. You will see the forwarder configuration page.
3. Modify the forwarder configuration as needed. When you're done, select **Deploy** to re-deploy the forwarder with the updated configuration. The forwarder instances will be re-deployed on a rolling basis over the next few minutes.

## Rename a forwarder

To rename a forwarder:

1. Click **Details** under the destination you want to change to open the destination details page.
2. On the event forwarders overview table, click the three dots next to the forwarder you want to change and select **Rename**.
3. Enter a new forwarder name and select **Rename** to save.

## Clone a forwarder

When creating a new forwarder, you can import the configuration from an existing forwarder of the same type. This is especially helpful when migrating a forwarder setup from a development pipeline to production.

To clone a forwarder:

1. Click **Add destination** and select the destination from the list of available integrations to start setting up the forwarder.
2. Give the forwarder a **name**, select the **pipeline** you want the forwarder to read events from, and choose a **connection**.
3. From the **Import configuration from** dropdown, choose an existing forwarder.
4. Click **Continue**. The filters, mappings, and custom functions will be pre-populated with those of the existing forwarder you imported from.

## Delete a forwarder

To permanently delete a forwarder:

1. In the event forwarders overview table, click the three dots next to the forwarder you want to change and select **Delete**.
2. On the confirmation modal, select **Delete**. This will start the process of destroying the underlying forwarder infrastructure.

---

# Event forwarding filter and mapping reference
> Complete reference for Snowplow event forwarding JavaScript expressions, field mapping syntax, event filtering, data transformations, and custom functions.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/reference/

Event forwarders use JavaScript expressions for filtering events and mapping Snowplow data to destination fields. These expressions are entered during [forwarder setup](/docs/destinations/forwarding-events/#getting-started) in Console, specifically in the **Event filtering**, **Field mapping**, and **Custom functions** sections. This reference covers the syntax and available data for these operations.

## Available event fields

You can reference any field in your Snowplow events for both filters and field mappings.

### Standard atomic fields

Access [standard Snowplow fields](/docs/fundamentals/canonical-event/) in your filters and mappings using JavaScript dot notation:

```javascript
// Standard atomic fields

event.app_id

event.event_name

event.platform

event.collector_tstamp

event.event_id

event.domain_userid

event.user_id

event.page_url

event.page_title

event.useragent

event.network_userid
```

### Custom events and entities

You can also access fields in Snowplow or custom event and entity schemas.

Forwarders transform Iglu schema URIs to JavaScript-safe field names:

| Original schema                            | Transformed field name               |
| ------------------------------------------ | ------------------------------------ |
| `com.acme/signup/jsonschema/1-0-0`         | `unstruct_event_com_acme_signup_1`   |
| `com.acme/user_profile/jsonschema/2-1-0`   | `contexts_com_acme_user_profile_2`   |
| `nl.basjes/yauaa_context/jsonschema/1-0-4` | `contexts_nl_basjes_yauaa_context_1` |

Schema names follow these transformation rules:

- Self-describing events: `unstruct_event_` prefix
- Entities: `contexts_` prefix
- Dots and slashes become underscores
- Only major version number retained
- Hyphens in vendor/name become underscores

> **Info:** Always use [optional chaining](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) (`?.`) when accessing custom events and entities to handle cases where they're not present.

For a self-describing event with schema `com.acme/signup/jsonschema/1-0-0`:

```javascript
// Access event properties

event?.unstruct_event_com_acme_signup_1?.signup_method

event?.unstruct_event_com_acme_signup_1?.user_type
```

For entities with schema `com.acme/user_profile/jsonschema/1-0-0`:

```javascript
// Access entity properties (entities are arrays)

event?.contexts_com_acme_user_profile_1?.[0]?.subscription_tier

event?.contexts_com_acme_user_profile_1?.[0]?.account_created
```

The field names above hardcode a schema's major version (the `_1` suffix), so an expression breaks when you start sending a new major version. To access custom events and entities across all versions, use the `allUnstruct` and `allContexts` [helper functions](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/#helper-functions).

`allUnstruct(event, eventName)` returns the payload of a self-describing event, or `undefined` if it's not present. The `eventName` is the snake\_case vendor and name, without the `unstruct_event_` prefix or version suffix:

```javascript
// Version-specific access:

event?.unstruct_event_com_acme_signup_1?.signup_method



// Version-agnostic alternative:

allUnstruct(event, "com_acme_signup")?.signup_method
```

`allContexts(event, entityName)` returns an array of all instances of an entity across schema versions, ordered with the highest version first. It returns an empty array if none are present:

```javascript
// Version-specific access:

event?.contexts_com_acme_user_profile_1?.[0]?.subscription_tier



// Version-agnostic alternative:

allContexts(event, "com_acme_user_profile")[0]?.subscription_tier
```

Other [helper functions](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/#helper-functions), such as `hash` and `parseTimestamp`, are also available in filters, field mappings, and custom functions.

## Event filtering

Event filters determine which events are forwarded to your destination. Only events matching your filter criteria (JavaScript expression evaluating to `true`) will be processed and sent.

### Basic filters

Filter events can contain standard JavaScript comparison operators:

```javascript
// Single condition

event.app_id == "website"



// Multiple conditions with AND

event.app_id == "website" && event.event == "page_view"



// Multiple conditions with OR

event.event_name == "add_to_cart" || event.event_name == "purchase"



// Check if event is in a list

["page_view", "add_to_cart", "purchase"].includes(event.event_name)



// Exclude events

event.app_id == "website" && event.event_name != "link_click"
```

### Advanced filtering patterns

Regular expressions:

```javascript
// Match multiple domains

event.page_urlhost.match(/mysite\.(com|fr|de)/)



// Match event name patterns

event.event_name.match(/^purchase_/)



// Match custom field patterns

event?.unstruct_event_com_acme_product_view_1?.category.match(/electronics|computers/)
```

Define reusable logic in the Custom Functions section:

```javascript
// Defined in the Custom Function editor panel

function isHighValueUser(event) {

  const profile = event?.contexts_com_acme_user_profile_1?.[0];

  return profile?.subscription_tier == "premium" ||

         profile?.lifetime_value > 1000;

}



// Use in filter

isHighValueUser(event) && event.event_name == "purchase"
```

## Field mapping

Field mapping defines how Snowplow event data is transformed and sent to destination APIs. Each mapping consists of:

- A destination field name (key)
- A JavaScript expression that extracts the value from your Snowplow event (value)

> **Info:** The code snippets below contain JavaScript expressions that you can include in the **Snowplow expression** mapping field in the UI.

### Basic mappings

Map standard event fields directly:

![Field mapping UI showing one row: the required destination field event\_type (described as \&quot;A unique identifier for your event\&quot;) mapped to the Snowplow expression event.event\_name.](/assets/images/event-forwarding-basic-mapping-be50ef0162cd510af04538089519f6ee.png)

```json
// sample output

{ "event_type": "page_view" }
```

You can also apply fallback and conditional logic:

![Field mapping UI showing two rows: user\_id mapped to event.userid ?? event.domain\_userid (nullish coalescing fallback), and source mapped to event.app\_id === \&quot;web\&quot; ? \&quot;website\&quot; : \&quot;app\&quot; (ternary conditional).](/assets/images/event-forwarding-conditional-mapping-b21729fb908ecd1cf784bb8f2952e6f9.png)

```json
// sample output

{

  "user_id": "a50d3dfe-ba21-432e-a165-1a1d2d633693"

  "source": "website"

}
```

You can also send static values:

![Field mapping UI showing one row: destination field source mapped to the static string value \&quot;snowplow\&quot;.](/assets/images/event-forwarding-static-mapping-89a993963e026dc4e9fc4a6bc421757c.png)

```json
// sample output

{ "source": "snowplow" }
```

### Data transformation

Convert data types, such as strings, boolean values, and dates:

![Field mapping UI showing four type conversion examples: page\_width mapped to parseInt(event.page\_width), page\_height to parseFloat(event.page\_height), is\_mobile to event.mobile\_device == \&quot;1\&quot;, and timestamp to new Date(event.collector\_tstamp).toISOString().](/assets/images/event-forwarding-type-conversions-cfc98aa6623258f0d8ec29cd32ef2576.png)

```json
// sample output

{

  "page_width": 720,

  "page_height": 600,

  "is_mobile": true,

  "timestamp": "2025-10-01T18:35:38.563Z"

}
```

Use standard [Javascript String methods](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String#instance_methods) to manipuate strings:

```javascript
// Case conversion

event.event_name.toLowerCase()

event.page_title.toUpperCase()



// String operations

event.page_url.replace("http://", "https://")

event.page_title.substring(0, 100)

event.page_urlpath.split('/')
```

Map to nested objects using dot notation in field names:

![Field mapping UI showing four nested object mappings using dot notation: user.id to event.domain\_userid, properties.page\_title to event.page\_title, properties.page\_url to event.page\_url, and properties.referrer to event.refr\_urlhost.](/assets/images/event-forwarding-nested-mapping-9f700b7f85d97a5edfe10bc78f9ae348.png)

```json
// sample output

{

  "user": {

    "id": "user123"

  },

  "properties": {

    "page_title": "Home Page",

    "page_url": "https://example.com",

    "referrer": "google.com"

  }

}
```

### Custom mapping functions

Define complex transformations as functions in the custom functions section. You can then reference these functions in filters and mappings. Below are a few example transformations:

```javascript
// Event name formatting

function formatEventName(event) {

  const nameMap = {

    'page_view': 'Page Viewed',

    'add_to_cart': 'Product Added to Cart',

    'purchase': 'Purchase Completed'

  };

  return nameMap[event.event_name] || event.event_name;

}



// Extract product data

function extractProductInfo(event) {

  const product = event?.unstruct_event_com_acme_product_1;

  if (!product) return null;



  return {

    id: product.product_id,

    name: product.product_name,

    category: product.category,

    price: parseFloat(product.price),

    currency: product.currency || 'USD'

  };

}



// User profile enrichment

function buildUserProfile(event) {

  const session = event?.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0];

  const geo = event?.contexts_com_snowplowanalytics_snowplow_geolocation_context_1?.[0];



  return {

    user_id: event.domain_userid,

    session_id: session?.sessionId,

    location: geo ? `${geo.latitude},${geo.longitude}` : null,

    user_agent: event.useragent,

    platform: event.platform

  };

}
```

## Other common patterns

Timestamp formatting:

```javascript
// ISO 8601 format

new Date(event.collector_tstamp).toISOString()



// Unix timestamp

Math.floor(new Date(event.collector_tstamp).getTime() / 1000)



// Readable format

new Date(event.collector_tstamp).toLocaleDateString()
```

Conditional field mapping:

```javascript
// Platform-specific mapping

event.platform == "web" ? event.page_url : event.screen_name



// Event-type specific properties

event.event_name == "purchase" ?

  event.unstruct_event_com_acme_purchase_1?.total_value :

  null
```

Array handling:

```javascript
// Get first entity

event?.contexts_com_acme_product_1?.[0]?.product_name



// Map all entities

event?.contexts_com_acme_product_1?.map(p => p.product_id)



// Filter and transform entities

event?.contexts_com_acme_product_1

  ?.filter(p => p.price > 10)

  ?.map(p => ({ id: p.product_id, name: p.product_name }))
```

---

# Data destinations overview
> Send Snowplow data to warehouses, lakes, and third-party platforms with event forwarding, native integrations, and reverse ETL for data activation.
> Source: https://docs.snowplow.io/docs/destinations/

Read more about Snowplow data destinations [here](/docs/fundamentals/destinations/).

[Data warehouses and data lakes](/docs/destinations/warehouses-lakes/) are primary destinations for Snowplow data.

Snowplow also supports many ways to [forward events](/docs/destinations/forwarding-events/) to a variety of platforms. Snowplow has native integrations as well as supporting Snowplow, vendor and community authored destinations via Google Tag Manager Server Side.

Finally, [reverse ETL](/docs/destinations/reverse-etl/) enables you to publish data from your warehouse directly to marketing platforms, where it can be activated.

---

# Reverse ETL for data activation
> Use reverse ETL to sync warehouse data to marketing and advertising platforms, activating audience segments built from Snowplow behavioral data.
> Source: https://docs.snowplow.io/docs/destinations/reverse-etl/

Reverse ETL tools sync data from your warehouse to external platforms such as Salesforce, Google Ads, and marketing automation tools. Combined with Snowplow's behavioral data, this lets you activate warehouse-derived audiences, scores, and attributes directly in the tools where your teams work.

Without reverse ETL, data in your warehouse is only available to tools that query it directly. Reverse ETL bridges that gap by continuously pushing selected data, such as customer segments or propensity scores, to downstream systems where it can be used for targeting, personalization, and outreach.

![Diagram of a marketing activation use case showing five stages: Collect (Snowplow strength), Unify, Segment, and Predict (data warehouse and models strength), and Activate (Reverse ETL strength)](/assets/images/reverseetl-70742060395d2f5b62046145cceaf1e2.png)

Common use cases include:

- Syncing propensity-to-buy scores to an ad platform to adjust bidding or exclude users who have already converted
- Pushing predicted lifetime value to a CRM to prioritize outreach
- Sending audience membership flags to an email platform for lifecycle campaigns

Snowplow recommends [Fivetran Activations](https://www.fivetran.com/data-movement/activations) for reverse ETL.

---

# Load Snowplow data to BigQuery
> Send Snowplow data to BigQuery for analytics and data warehousing with automatic table creation, schema evolution, and cross-batch deduplication.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/bigquery/

> **Info:** The BigQuery integration is available for Snowplow pipelines running on **AWS** and **GCP**.

The Snowplow BigQuery integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) directly into your BigQuery datasets for analytics, data modeling, and more.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

- Provide your Google Cloud Project ID and region
- Allow-list Snowplow IP addresses
- Specify the desired dataset name
- Create a service account with the `roles/bigquery.dataEditor` permission (more permissions will be required for loading failed events and setting up [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))

## Getting started

You can add a BigQuery destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **BigQuery**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **BigQuery**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

> **Tip:** For more details on the loading flow, see the [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in BigQuery

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                |
| ------- | --- | ------------------------------------------------------- | -------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `RECORD`) | data for your custom `product` entities (as `REPEATED RECORD`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=bigquery) Snowplow data.

---

# Load Snowplow data to Databricks
> Send Snowplow data to Databricks for analytics and data processing with Delta Lake tables, schema evolution, and lakehouse architecture support.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/databricks/

> **Info:** The Databricks integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

The Snowplow Databricks integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into your Databricks environment for analytics, data modeling, and more.

Depending on the cloud provider for your Snowplow pipeline, there are different options for this integration:

| Integration                                                                                                         | AWS | Azure | GCP | Failed events support |
| ------------------------------------------------------------------------------------------------------------------- | --- | ----- | --- | --------------------- |
| Direct, batch-based ([RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/))                | ✅   | ❌     | ❌   | ❌                     |
| Via Delta Lake ([Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/))                            | ❌¹  | ✅²    | ✅²  | ✅                     |
| Streaming / Lakeflow ([Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/)) | ✅   | ✅     | ✅   | ✅                     |

_¹ Delta+Databricks combination is currently not supported for AWS pipelines. The loader uses DynamoDB tables for mutually exclusive writes to S3, a feature of Delta. Databricks, however, does not support this (as of September 2025). This means that it’s not possible to alter the data via Databricks (e.g. to run `OPTIMIZE` or to delete PII)._

_² The lake must be in the same cloud as the pipeline._

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to do a few things.

**Batch-based (AWS):**

- Provide a Databricks cluster along with its URL

- Specify the Unity catalog name and schema name

- Create an access token with the following permissions:

  - `USE CATALOG` on the catalog
  - `USE SCHEMA` and `CREATE TABLE` on the schema
  - `CAN USE` on the SQL warehouse

**Via Delta Lake (Azure, GCP):**

See [Delta Lake](/docs/destinations/warehouses-lakes/delta/).

**Streaming:**

- Create an S3 or GCS bucket or ADLS storage container, located in the same cloud and region as your Databricks instance

- Create a storage credential to allow Databricks to access the bucket or container

- Create an external location and a volume within Databricks pointing to the above

- Provide a Databricks SQL warehouse URL, Unity catalog name and schema name

- Create a service principal and grant the following permissions:

  - `USE CATALOG` on the catalog
  - `USE SCHEMA` and `CREATE TABLE` on the schema
  - `READ VOLUME` and `WRITE VOLUME` on the volume
  - `CAN USE` on the SQL warehouse (for testing the connection and monitoring, e.g. as part of the [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))

Note that Lakeflow features require a Premium Databricks account. You might also need Databricks metastore admin privileges for some of the steps.

***

## Getting started

You can add a Databricks destination through the Snowplow Console.

**Batch-based (AWS):**

(For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Databricks**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Databricks**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

**Via Delta Lake (Azure, GCP):**

Follow the instructions for [Delta Lake](/docs/destinations/warehouses-lakes/delta/#getting-started).

Then create an external table in Databricks pointing to the Delta Lake location.

**Streaming:**

(For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Databricks Streaming**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Databricks**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

Once the loader is up and running, click on the “...” button in the **Loaders** table and select **Databricks setup instructions**. Follow the outlined steps to create a Lakeflow pipeline within Databricks.

***

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

**Batch-based (AWS):**

For more details on the loading flow, see the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) reference page, where you will find additional information and diagrams.

**Via Delta Lake (Azure, GCP):**

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

**Streaming:**

For more details on the loading flow, see the [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) reference page, where you will find additional information and diagrams.

***

## Snowplow data format in Databricks

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data.

---

# Load Snowplow data to Delta Lake
> Send Snowplow data to Delta Lake for analytics and data processing with ACID transactions, schema evolution, and time travel capabilities.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/delta/

> **Info:** The Delta Lake integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

Delta Lake is an open table format for data lake architectures. The Snowplow Delta integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into Delta tables in your data lake for analytics, data modeling, and more.

Data in Delta Lake can be consumed using various tools and products, for example:

- Amazon Athena
- Apache Spark or Amazon EMR
- Databricks¹
- Microsoft Synapse Analytics
- Microsoft Fabric

_¹ Delta+Databricks combination is currently not supported for AWS pipelines. The loader uses DynamoDB tables for mutually exclusive writes to S3, a feature of Delta. Databricks, however, does not support this (as of September 2025). This means that it’s not possible to alter the data via Databricks (e.g. to run `OPTIMIZE` or to delete PII)._

> **Note:** Currently, we only support loading to a lake in the same cloud as your Snowplow pipeline.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

**AWS:**

- Provide an S3 bucket

- Create a DynamoDB table (required for file locking)

- Create an IAM role with the following permissions:

  - For the S3 bucket:

    - `s3:ListBucket`
    - `s3:GetObject`
    - `s3:PutObject`
    - `s3:DeleteObject`
    - `s3:ListBucketMultipartUploads`
    - `s3:AbortMultipartUpload`

  - For the DynamoDB table:

    - `dynamodb:DescribeTable`
    - `dynamodb:Query`
    - `dynamodb:Scan`
    - `dynamodb:GetItem`
    - `dynamodb:PutItem`
    - `dynamodb:UpdateItem`
    - `dynamodb:DeleteItem`

- Schedule a regular job to optimize the lake

**GCP:**

- Provide a GCS bucket
- Create a service account with the `roles/storage.objectUser` role on the bucket
- Create and provide a service account key

**Azure:**

- Provide an ADLS storage container
- Create a new App Registration with the `Storage Blob Data Contributor` permission
- Provide the registration tenant ID, client ID and client secret

***

## Getting started

You can add a Delta Lake destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/lake-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Delta**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Delta**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

We recommend scheduling regular [lake maintenance jobs](/docs/api-reference/loaders-storage-targets/lake-loader/maintenance/?lake-format=delta) to ensure the best long-term performance.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Delta Lake

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data. (You will need a query engine such as Spark SQL or Databricks to query Delta tables.)

---

# Load Snowplow data to Apache Iceberg
> Send Snowplow data to Apache Iceberg data lakes for analytics and data processing with open table format, schema evolution, and cross-engine compatibility.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/iceberg/

> **Info:** The Iceberg integration is available for Snowplow pipelines running on **AWS** and **GCP** only.

Apache Iceberg is an open table format for data lake architectures. The Snowplow Iceberg integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into Iceberg tables in your data lake for analytics, data modeling, and more.

Iceberg data can be consumed using various tools and products, for example:

- Amazon Athena
- Amazon Redshift Spectrum
- Apache Spark or Amazon EMR
- Snowflake
- ClickHouse

We currently support the following catalogs:

| Catalog | AWS | GCP |
| ------- | --- | --- |
| Glue    | ✅   | ❌   |
| REST¹   | ✅   | ✅   |

_¹The REST catalog has only been tested with the Snowflake Open Catalog implementation._

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

**REST:**

- Specify your Snowflake Open Catalog account id and region, as well as namespace
- Create a service connection to the catalog and provide the client id and client secret

**AWS Glue:**

- Specify your AWS account ID

- Provide an S3 bucket and an AWS Glue database

- Create an IAM role with the following permissions:

  - For the S3 bucket:

    - `s3:ListBucket`
    - `s3:GetObject`
    - `s3:PutObject`
    - `s3:DeleteObject`

  - For the Glue database:

    - `glue:CreateTable`
    - `glue:GetTable`
    - `glue:UpdateTable`

- Schedule a regular job to optimize the lake

***

## Getting started

You can add an Iceberg destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/lake-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Iceberg**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Iceberg**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

For AWS Glue, we recommend scheduling regular [lake maintenance jobs](/docs/api-reference/loaders-storage-targets/lake-loader/maintenance/?lake-format=iceberg) to ensure the best long-term performance.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Iceberg

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data. (You will need a query engine such as Spark SQL or Snowflake to query Iceberg tables.)

---

# Supported warehouse and data lake destinations
> An overview of the available options for storing Snowplow data in data warehouses and lakes
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/

Data warehouses and data lakes are primary destinations for Snowplow data. For other options, see the [destinations overview](/docs/fundamentals/destinations/) page.

### Data warehouses

- [Snowflake](/docs/destinations/warehouses-lakes/snowflake/)
- [Databricks](/docs/destinations/warehouses-lakes/databricks/)
- [BigQuery](/docs/destinations/warehouses-lakes/bigquery/)
- [Redshift](/docs/destinations/warehouses-lakes/redshift/)

### Data lakes

- [Iceberg](/docs/destinations/warehouses-lakes/iceberg/)
- [Delta Lake](/docs/destinations/warehouses-lakes/delta/)

---

# How to query Snowplow data in the warehouse
> Introduction to querying Snowplow data in warehouses including self-describing events, entities, and handling duplicate events with SQL techniques.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/querying-data/

You will typically find most of your Snowplow data in the `events` table. If you are using Redshift, there will be extra tables for [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) — see [below](#self-describing-events).

Please refer to [the structure of Snowplow data](/docs/fundamentals/canonical-event/) for the principles behind our approach, as well as the descriptions of the various standard columns.

> **Tip:** Querying the `events` table directly can be useful for exploring your events or building custom analytics. However, for many common use cases it’s much easier to use our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/), which provide a pre-aggregated view of your data.

The simplest query could look like this:

```sql
SELECT * FROM <events>

WHERE event_name = 'page_view'
```

You will need to replace `<events>` with the appropriate location — the database, schema and table name will depend on your configuration.

> **Note:** With large data volumes (read: any production system), you should always include a filter on the partition key (normally, `collector_tstamp`), for example:
>
> ```sql
> WHERE ... AND collector_tstamp between timestamp '2023-10-23' and timestamp '2023-11-23'
> ```
>
> This ensures that you read from the minimum number of (micro-)partitions necessary, making the query run much faster and reducing compute cost (where applicable).

## Self-describing events

[Self-describing events](/docs/fundamentals/events/#self-describing-events) can contain their own set of fields, defined by their [schema](/docs/fundamentals/schemas/).

**Redshift:**

For Redshift users, self-describing events are not part of the standard `events` table. Instead, each type of event is in its own table. The table name and the fields in the table will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query just the table for that particular self-describing event, if that's all that's required for your analysis, or join that table back to the `events` table:

```sql
SELECT

    ...

FROM

    <schema>.<events> ev

LEFT JOIN

    <schema>.my_example_event_table sde

    ON sde.root_id = ev.event_id AND sde.root_tstamp = ev.collector_tstamp
```

> **Note:** You may need to take care of [duplicate events](#dealing-with-duplicates).

**BigQuery:**

Each type of self-describing event is in a dedicated `RECORD`-type column. The column name and the fields in the record will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event like so:

```sql
SELECT

    ...

    unstruct_event_my_example_event_1.my_field,

    ...

FROM

    <events>
```

> **Note:** The column name produced by previous versions of the BigQuery Loader (<2.0.0) would contain full schema version, e.g. `unstruct_event_my_example_event_1_0_0`. The [BigQuery Loader upgrade guide](/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/2-0-0-upgrade-guide/) describes how to enable the legacy column names in the 2.0.0 loader.

**Snowflake:**

Each type of self-describing event is in a dedicated `OBJECT`-type column. The column name will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event like so:

```sql
SELECT

    ...

    unstruct_event_my_example_event_1:myField::varchar, -- field will be variant type so important to cast

    ...

FROM

    <events>
```

**Databricks, Spark SQL:**

Each type of self-describing event is in a dedicated `STRUCT`-type column. The column name and the fields in the `STRUCT` will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event by extracting them like so:

```sql
SELECT

    ...

    unstruct_event_my_example_event_1.my_field,

    ...

FROM

    <events>
```

**Synapse Analytics:**

Each type of self-describing event is in a dedicated column in JSON format. The column name will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event like so:

```sql
SELECT

    ...

    JSON_VALUE(unstruct_event_my_example_event_1, '$.my_field')

    ...

FROM

    OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') AS events
```

***

## Entities

[Entities](/docs/fundamentals/entities/) (also known as contexts) provide extra information about the event, such as data describing a product or a user.

**Redshift:**

For Redshift users, entities are not part of the standard `events` table. Instead, each type of entity is in its own table. The table name and the fields in the table will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

The entities can be joined back to the core `events` table by the following, which is a one-to-one join (for a single record entity) or a one-to-many join (for a multi-record entity), assuming no duplicates.

```sql
SELECT

    ...

FROM

    <schema>.<events> ev

LEFT JOIN -- assumes no duplicates, and will return all events regardless of if they have this entity

    <schema>.my_entity ent

    ON ent.root_id = ev.event_id AND ent.root_tstamp = ev.collector_tstamp
```

> **Note:** You may need to take care of [duplicate events](#dealing-with-duplicates).

**BigQuery:**

Each type of entity is in a dedicated `REPEATED RECORD`-type column. The column name and the fields in the record will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT

    ...

    contexts_my_entity_1[SAFE_OFFSET(0)].my_field AS my_field,

    ...

FROM

    <events>
```

Alternatively, you can use the [`unnest`](https://cloud.google.com/bigquery/docs/reference/standard-sql/arrays#flattening_arrays) function to explode out the array into one row per entity value.

```sql
SELECT

    ...

    my_ent.my_field AS my_field,

    ...

FROM

    <events>

LEFT JOIN

    unnest(contexts_my_entity_1) AS my_ent -- left join to avoid discarding events without values in this entity
```

> **Note:** Column name produced by previous versions of the BigQuery Loader (<2.0.0) would contain full schema version, e.g. `contexts_my_entity_1_0_0`.

**Snowflake:**

Each type of entity is in a dedicated `ARRAY`-type column. The column name will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT

    ...

    contexts_my_entity_1[0]:myField::varchar,  -- field will be variant type so important to cast

    ...

FROM

    <events>
```

Alternatively, you can use the [`lateral flatten`](https://docs.snowflake.com/en/sql-reference/functions/flatten) function to explode out the array into one row per entity value.

```sql
SELECT

    ...

    r.value:myField::varchar,  -- field will be variant type so important to cast

    ...

FROM

    <events> AS t,

    LATERAL FLATTEN(input => t.contexts_my_entity_1) r
```

**Databricks, Spark SQL:**

Each type of entity is in a dedicated `ARRAY<STRUCT>`-type column. The column name and the fields in the `STRUCT` will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT

    ...

    contexts_my_entity_1[0].my_field,

    ...

FROM

    <events>
```

Alternatively, you can use the [`LATERAL VIEW`](https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-lateral-view.html) clause combined with [`EXPLODE`](https://docs.databricks.com/sql/language-manual/functions/explode.html) to explode out the array into one row per entity value.

```sql
SELECT

    ...

    my_ent.my_field,

    ...

FROM

    <events>

    LATERAL VIEW EXPLODE(contexts_my_entity_1) AS my_ent
```

**Synapse Analytics:**

Each type of entity is in a dedicated column in JSON format. The column name will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT

    ...

    JSON_VALUE(contexts_my_entity_1, '$[0].my_field')

    ...

FROM

    OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') AS events
```

Alternatively, you can use the [`CROSS APPLY` clause combined with `OPENJSON`](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-parquet-nested-types#project-values-from-repeated-columns) to explode out the array into one row per entity value.

```sql
SELECT

    ...

    JSON_VALUE(my_ent.[value], '$.my_field')

    ...

FROM

    OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') as events

    CROSS APPLY OPENJSON(contexts_my_entity_1) AS my_ent
```

***

## Failed events

See [Exploring failed events](/docs/monitoring/exploring-failed-events/).

## Dealing with duplicates

In some cases, your data might contain duplicate events (full deduplication _before_ the data lands in the warehouse is optionally available for [Redshift, Snowflake and Databricks on AWS](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/)).

While our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) deal with duplicates for you, there may be cases where you need to de-duplicate the events table yourself.

**Redshift:**

In Redshift, you must first generate a `ROW_NUMBER()` on your events and use this to de-duplicate.

```sql
WITH unique_events AS (

    SELECT

        ...

        ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) AS event_id_dedupe_index

    FROM

        <events> a

)



SELECT

    ...

FROM

    unique_events

WHERE

    event_id_dedupe_index = 1
```

Things get a little more complicated if you want to join your event data with a table containing [entities](#entities).

Suppose your entity is called `my_entity`. If you know that each of your events has at most 1 such entity attached, the de-duplication requires the use of a row number over `event_id` to get each unique event:

```sql
WITH unique_events AS (

    SELECT

        ev.*,

        ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) AS event_id_dedupe_index

    FROM

        <schema>.<events> ev

),



unique_my_entity AS (

    SELECT

        ent.*,

        ROW_NUMBER() OVER (PARTITION BY a.root_id ORDER BY a.root_tstamp) AS my_entity_index

    FROM

        <schema>.my_entity_1 ent

)



SELECT

    ...

FROM

    unique_events u_ev

LEFT JOIN

    unique_my_entity u_ent

    ON u_ent.root_id = u_ev.event_id AND u_ent.root_tstamp = u_ev.collector_tstamp AND u_ent.my_entity_index = 1

WHERE

    u_ev.event_id_dedupe_index = 1
```

If your events might have more than one `my_entity` attached, the logic is slightly more complex.

**Details**

First, de-duplicate the events table in the same way as above, but also keep track of the number of duplicates (see `event_id_dedupe_count` below). In the entity table, generate a row number per unique combination of _all_ fields in the record. Then join on `root_id` and `root_tstamp` as before, but with an _additional_ clause that the row number is a multiple of the number of duplicates, to support the 1-to-many join. This ensures all duplicates are removed while retaining all original records of the entity. This may look like a weird join condition, but it works.

Unfortunately, listing all fields manually can be quite tedious, but we have added support for this in the [de-duplication logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/) of our dbt packages.

```sql
WITH unique_events AS (

    SELECT

        ev.*,

        ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) AS event_id_dedupe_index,

        COUNT(*) OVER (PARTITION BY a.event_id) AS event_id_dedupe_count

    FROM

        <schema>.<events> ev

),



unique_my_entity AS (

    SELECT

        ent.*,

        ROW_NUMBER() OVER (PARTITION BY a.root_id, a.root_tstamp, ... /*all columns listed here for your entity */ ORDER BY a.root_tstamp) AS my_entity_index

    FROM

        <schema>.my_entity_1 ent

)



SELECT

    ...

FROM

    unique_events u_ev

LEFT JOIN

    unique_my_entity u_ent

    ON u_ent.root_id = u_ev.event_id AND u_ent.root_tstamp = u_ev.collector_tstamp AND mod(u_ent.my_entity_index, u_ev.event_id_dedupe_count) = 0

WHERE

    u_ev.event_id_dedupe_index = 1
```

**BigQuery:**

In BigQuery it is as simple as using a `QUALIFY` statement over your initial query:

```sql
SELECT

    ...

FROM <events> a

QUALIFY ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) = 1
```

**Snowflake:**

In Snowflake it is as simple as using a `qualify` statement over your initial query:

```sql
SELECT

    ...

FROM <events> a

QUALIFY ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) = 1
```

**Databricks, Spark SQL:**

In Databricks it is as simple as using a `qualify` statement over your initial query:

```sql
SELECT

    ...

FROM <events> a

QUALIFY ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) = 1
```

**Synapse Analytics:**

In Synapse you must first generate a `ROW_NUMBER()` on your events and use this to de-duplicate.

```sql
WITH unique_events AS (

    SELECT

        ...

        ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY collector_tstamp) AS event_id_dedupe_index

    FROM

        OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') AS events

)



SELECT

    ...

FROM

    unique_events

WHERE

    event_id_dedupe_index = 1
```

***

---

# Load Snowplow data to Amazon Redshift
> Send Snowplow data to Amazon Redshift for analytics and data warehousing with automatic table creation, schema evolution, and optimized batch loading from S3.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/redshift/

> **Info:** The Redshift integration is available for Snowplow pipelines running on **AWS** only.

The Snowplow Redshift integration allows you to load enriched event data directly into your Redshift cluster (including Redshift serverless) for analytics, data modeling, and more.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

- Provide your Redshift cluster endpoint and connection details

- Allow-list Snowplow IP addresses

- Specify the desired database and schema names

- Create a user and a role with the following permissions:

  - Schema ownership (`CREATE SCHEMA ... AUTHORIZATION`)
  - `SELECT` on system tables (`svv_table_info`, `svv_interleaved_columns`, `stv_interleaved_counts`) — this is required for maintenance jobs

## Getting started

You can add a Redshift destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Redshift**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Redshift**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Redshift

The event data is split across multiple tables.

The main table (`events`) contains the [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For each type of self-describing event and entity, there are additional tables that can be joined with the main table:

**com\_acme\_button\_press\_1**

| root\_id                             | root\_tstamp            | button\_name | button\_color | ... |
| ------------------------------------ | ----------------------- | ------------ | ------------- | --- |
| c6ef3124-b53a-4b13-a233-0088f79dcbcb | 2025-05-06 12:30:05.123 | Cancel       | red           | ... |

**com\_acme\_product\_1**

| root\_id                             | root\_tstamp            | name   | price | ... |
| ------------------------------------ | ----------------------- | ------ | ----- | --- |
| c6ef3124-b53a-4b13-a233-0088f79dcbcb | 2025-05-06 12:30:05.123 | Salt   | 2.60  | ... |
| c6ef3124-b53a-4b13-a233-0088f79dcbcb | 2025-05-06 12:30:05.123 | Pepper | 3.10  | ... |

> **Note:** The `_1` suffix represents the major version of the schema (e.g. `1-x-y`).

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=redshift) Snowplow data.

---

# Load Snowplow data to Snowflake
> Send Snowplow data to Snowflake for analytics and data warehousing with automatic table management, schema evolution, and efficient data loading via Snowpipe or batch.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/snowflake/

> **Info:** The Snowflake integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

The Snowplow Snowflake integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) directly into your Snowflake warehouse for analytics, data modeling, and more.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

- Provide your Snowflake account locator URL, cloud provider and region

- Allow-list Snowplow IP addresses

- Generate a key pair for key-based authentication

- Specify the desired database and schema names, as well as a warehouse name

- Create a role with the following permissions:

  - `USAGE`, `OPERATE` on warehouse (for testing the connection and monitoring, e.g. as part of the [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))
  - `USAGE` on database
  - `ALL` privileges on the target schema

## Getting started

You can add a Snowflake destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Snowflake**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Snowflake**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Snowflake

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1                      | contexts\_com\_acme\_product\_1                                |
| ------- | --- | ----------------------------------------------------------------- | -------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as a `VARIANT` object) | data for your custom `product` entities (as a `VARIANT` array) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=snowflake) Snowplow data.

---

# Data structure management overview
> Create, manage, and update data structures (schemas) using Snowplow Console UI, Data Structures API, Snowplow CLI, or Iglu for community users.
> Source: https://docs.snowplow.io/docs/event-studio/data-structures/

This section explains how to create, manage, and update [data structures](/docs/fundamentals/schemas/) (schemas). Snowplow provides different options for data structure management:

- Snowplow Console UI
- [Data structures API](/docs/event-studio/programmatic-management/data-structures-api/)
- [Snowplow CLI](/docs/api-reference/snowplow-cli/)
- For Community users: [Iglu](/docs/api-reference/iglu/iglu-repositories/iglu-server/)

## Create a data structure

To create a data structure within Console, go to **Data collection** > **Data structures** and click the **Create data structure** button. You can use the builder to create simple data structures with basic types and validation rules, or the JSON editor for more complex data structures.

> **Info:** The data structure builder supports the following types:
>
> - String
> - Enumerated list
> - Integer
> - Decimal
> - Boolean
>
> For more complex data structures that require nesting or more advanced data types, use the [JSON editor](/docs/event-studio/data-structures/json-editor/). To understand all available JSON Schema validation options, see the [JSON Schema reference](/docs/api-reference/json-schema-reference/).

Populate the general information, such as Name, and a Description and Vendor. Vendor allows you to organize your data structures, for example, by teams. Snowplow will automatically generate the Tracking Url to be referenced in your tracking code.

![Data structure builder General Information form for a new data structure named checkout\_funnel, with type set to Event, vendor set to snowplow.demo, and an autogenerated schema tracking URL of iglu.demo/checkout\_funnel/jsonschema/1-0-0. The Properties section shows no properties added yet, with an Add a property button.](/assets/images/data-structures-1-5b794bc4236fadba42000ce5d714b43f.png)

When creating a new [data structure](/docs/fundamentals/schemas/), you can add one or multiple properties. For each property, you can set a name, description, its type and a possible enumeration of allowed values (for type `string`). You can also set additional constraints to define if this property should be optional or mandatory, and if `null` values are allowed.

![Create a Property dialog for the checkout\_funnel data structure, with property name action\_name, type String, string type set to Enumerated list with values add\_to\_cart, view\_cart, checkout, proceed\_to\_pay, and pick\_address, nullable set to No, and required set to Yes.](/assets/images/data-structures-2-b23aa8f4043bced0dfd2ac52eb0267d5.png)

Click **Save** on the Property dialog box to save your property changes.

Clicking **Save** on the data structure page will save your data structure as a draft. At this point, your data structure is not yet deployed to your development environment and cannot be used for event validation. When you're ready to test your data structure, you'll need to deploy it from the draft state to your development environment.

## Working with drafts

When you create a new data structure your changes are initially saved as a **draft**. Drafts allow you to:

- Make multiple changes without worrying about version numbers
- Experiment freely before committing to a final version
- Review and refine your data structure before deployment

**Important**: Draft data structures are not deployed to your development environment and will not be available for event validation. You must deploy your draft to the development environment when you're ready to test it.

This workflow gives you the flexibility to iterate on your data structure design without the overhead of managing version increments for every small change.

## Edit a data structure

To edit an existing data structure, navigate to **Data Structures** and locate the data structure you wish to edit. You can more easily find your data structure by:

- Using the search facility to search by name or vendor
- Ordering the Name column alphabetically
- Filtering the listing by Type and / or Vendor

Once located, click on the name to view the data structure. You can then select from two options to edit the data structure: **Edit with builder** or **Edit with JSON editor**.

> **Note:** The **Edit with builder** option will be unavailable if the data structure you're viewing is not supported. More complex data structures must be edited with the **JSON Editor**.

On the edit page, under the General Information panel, you can update the data structure type or its description. To add a new property, cick the "Add Property" button. To edit or delete an existing property, click the three dots next to the property name to open the action menu, and then select the appropriate option.

![Data structure builder edit page for cart\_action showing two properties in the Properties table: type (string, required, must be one of add or remove) and propertytwo (integer, not required). The three-dot action menu is open on the type property, showing Edit property and Delete property options.](/assets/images/edit-data-structure-a751c564691dd2386c85aae6a9702fb0.png)

When you modify the data structure, the builder will mark your changes in yellow, and automatically determine the new version of your data structure based on these modifications. You can reset the data structure and erase all changes at any moment by clicking the **Clear Changes** button found in the alert beneath the properties.

If you are satisfied with your changes, click **Save** and make sure to note the newly updated tracking URL.

![Data structure builder for cart\_action after edits, with the schema tracking URL highlighted in yellow showing the version incremented to 2-0-0. A yellow banner at the bottom reads \&quot;Version increased to 2-0-0 — your current version is 1-1-0, based on your changes the new version will be 2-0-0\&quot; with a Clear changes button.](/assets/images/data-structure-version-e2f76da4a575ceedae49575b78954288.png)

## Promote a data structure

When you're ready to use your data structure, you need to publish it from draft status to your development environment for testing.

Once you are happy with your changes in the development environment, you will want to migrate these changes to your production environment.

> **Note:** The action of migrating data structures to production is only available to Admin users.

Navigate to **Data structures** and locate the data structure you wish to migrate. You can more easily find your data structure by:

- Using the search facility to search by name or vendor
- Ordering the Name column alphabetically
- Filtering the listing by Type and / or Vendor

Once located, either click on the name to view the data structure and then click the "Migrate to production" button, or click the three dots to bring up the action menu where you can select "Migrate to production".

![Data structures listing filtered by the search term \&quot;user\&quot;, showing five results with Development and Production version columns. Two entries have their Development version highlighted in yellow (1-0-2 and 1-0-1), indicating versions not yet promoted. The three-dot action menu is open on the first result, showing options including View data structure, Edit JSON, Edit with builder, Migrate to production, and Hide data structure.](/assets/images/image-8-887f63361cd518905ef3ba0aa4294c88.png)

At this stage you will see the publish dialog, and depending on how you versioned your edits you will see one of two messages:

If you are **publishing a new schema**, or **have incremented** the version whilst editing then you will see a confirmation of the action. Click **Migrate to Production** to migrate the data structure.

If you **have patched** the version whilst editing then you will see a warning that you must increment before publishing. Patching the version on Production is not a permitted action. [Increment the version number according to the changes you have made](/docs/event-studio/data-structures/versioning/) and click **Migrate to production** to migrate the latest version of your data structure to your production environment.

Your data structure will now be available in your production environment to send events against.

## Hide a data structure

Sometimes you will make errors when creating a data structure, or simply be creating new data structures as part of a quick experiment. On these occasions you may wish to hide the schema to clean up the listing in Console.

Navigate to **Data structures** and locate the data structure you wish to hide. You can more easily find your data structure by:

- Using the search facility to search by name
- Ordering the Name column alphabetically
- Filtering the listing by Type and / or Vendor

Once located either click on the name to view the data structure and then click the **Hide** button, or click the three dots to bring up the action menu where you can select **Hide data structure**.

Follow the dialog instructions to confirm the action.

> **Note:** Hiding a data structure will not remove it from the registry, it simply hides it from the console listing. This means:
>
> (1) events can still be sent against this structure (2) you cannot create a new structure of the same name

### Restore a hidden data structure

If you have hidden a data structure and wish to restore it, navigate to the bottom of the list of data structures and locate the 'View hidden data structures' link.

![Bottom of the data structures listing showing two entries (user and tenant\_users) and a View hidden data structures link below the table.](/assets/images/image-9-aaac318a1695105606bcf8e362e3314a.png)

This will take you to a list of hidden data structures, locate the one you wish to restore and click **Restore data structure** to show it in the main listing.

## Externally managed data structures

Data structures can be managed from an external repository using [Snowplow CLI](/docs/api-reference/snowplow-cli/data-structures/).

When a data structure is managed this way it becomes locked in the UI, disabling all editing. You will see a banner explaining the situation and giving people with the 'publish to production' (default for admin users) capability the ability to unlock.

![Info banner reading \&quot;This data structure is locked because it is managed in an external repository and can\&#39;t be edited from the console UI. Repository: https://myrepo.org\&quot; with Unlock and Learn more links.](/assets/images/locked-ds-5318a0b425e253f84213ba78ba9743de.png)

> **Tip:** Having a single source of truth for a data structure is a good idea. If your source of truth is an external repository then unlocking and editing will cause conflicts.

---

# Create complex data structures with the JSON editor
> Define complex data structures with heavy nesting and advanced data types using the JSON Editor for full JSON Schema support.
> Source: https://docs.snowplow.io/docs/event-studio/data-structures/json-editor/

> **Info:** The JSON editor is ideal for more complex data structures that require nesting or more advanced data types. For simple data structures, use the [Data Structures Builder](/docs/event-studio/data-structures/).

## Creating a new data structure

Select whether you'd like to create an [Event](/docs/fundamentals/events/) or an [Entity](/docs/fundamentals/entities/). You can always change this selection at a later date.

![Choice between builder and JSON editor options](/assets/images/image-2-bad6698ed68cad8d2bd005ce7ff2a082.png)

You can now write the first version of your JSON schema for this data structure. Some template JSON is provided in the code window to start you off. For comprehensive guidance on all supported JSON Schema features and validation options, see the [JSON Schema reference](/docs/api-reference/json-schema-reference/).

![JSON editor \&quot;Create data structure\&quot; page showing a dark-themed code editor pre-populated with a JSON Schema template including description, properties, additionalProperties, type, required, self (vendor, name, format, version 1-0-0), and $schema fields. Validate and Exit without saving buttons are shown at the bottom.](/assets/images/json-template-406dba60e0fcee3f5c03ed6e579372d4.png)

Once you are done, click the **Validate** button and we'll validate that your schema is valid JSON markup. Assuming it passes validation, you can save your data structure as a draft.

See the [Working with drafts](/docs/event-studio/data-structures/#working-with-drafts) section for more information about the draft workflow.

Click **Save as draft** to save your data structure as a draft. As this is the first version of your data structure, it will be created as version `1-0-0` when you later deploy it to your development environment.

## Editing a data structure

Make the required edits to the JSON schema. You can use the 'Difference' toggle above the editor to see a 'diff' view against the latest Production version of your data structure.

In the example below we have changed the `maxLength` of `example_field_1`.

![Side-by-side diff view comparing \&quot;Latest production version (1-0-0)\&quot; on the left with \&quot;New version\&quot; on the right. Line 7 is highlighted: maxLength is changed from 128 (shown in red on the left) to 256 (shown in green on the right).](/assets/images/image-5-64bfbf5b5861f6d058d055bf896f5192.png)

Once you are happy with your changes, click **Validate** to ensure you have valid JSON markup. Then click **Publish to development environment** to save your changes to your development environment.

![Publish version to development environment dialog for the \&quot;enrichments\_workflow\&quot; schema showing three version options: Patch (1-0-1) — currently selected, Non-breaking (1-0-2), and Breaking (2-0-0). A warning banner advises selecting the correct version increment to avoid load failures.](/assets/images/image-7-9423f8ff7c3adcfd4e7d36858e60a934.png)

The versioning dialog will appear, at this point you have three options:

- Increment a minor version to indicate a non-breaking change to the schema. In our example, this would increment the schema to from `1-0-1` to `1-0-2`.
- Increment a major version to indicate a breaking change to the schema. In our example, this would increment the schema from `1-0-1` to `2-0-0`.
- [Patch the current version](/docs/event-studio/data-structures/versioning/#patch-a-schema), this will overwrite the existing schema without increasing the version. In our example, this would leave the schema at 1-0-1.

For more information see [Versioning your data structures](/docs/event-studio/data-structures/versioning/).

Once you have selected the appropriate version, click **Deploy to development environment** and your data structure will be deployed to your development environment ready [for you to test](/docs/testing/).

You can identify data structures where the Development version is ahead of the Production version by the yellow background on the version number. In this example both `user` and `alert` have been edited on development.

***

---

# Version and amend data structures
> Evolve your tracking design safely with backwards-compatible data structure versioning using JSON schema version numbers to control warehouse loader behavior.
> Source: https://docs.snowplow.io/docs/event-studio/data-structures/versioning/

Every data structure is based on a [versioned schema](/docs/fundamentals/schemas/versioning/).

There are two kinds of schema changes:

- **Non-breaking** — a non-breaking change is backward compatible with historical data and increments the `patch` version (for example, `1-0-0` to `1-0-1`) or the `minor` version (for example, `1-0-0` to `1-1-0`).
- **Breaking** — a breaking change is not backward compatible with historical data and increments the `major` version (for example, `1-0-0` to `2-0-0`).

Different data warehouses handle schema evolution slightly differently. Use the table below as a guide for incrementing the schema version appropriately.

|                                              | Redshift     | Snowflake, BigQuery, Databricks |
| -------------------------------------------- | ------------ | ------------------------------- |
| **Add / remove / rename an optional field**  | Non-breaking | Non-breaking                    |
| **Add / remove / rename a required field**   | Breaking     | Breaking                        |
| **Change a field from optional to required** | Breaking     | Breaking                        |
| **Change a field from required to optional** | Breaking     | Non-breaking                    |
| **Change the type of an existing field**     | Breaking     | Breaking                        |
| **Change the size of an existing field**     | Non-breaking | Non-breaking                    |

> **Note:** In Redshift and Databricks, changing _size_ may also mean _type_ change. For example, changing the `maximum` integer from `30000` to `100000`. See our documentation on [how schemas translate to database types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

## Automatic versioning with the data structure builder

Versioning is automated when using the data structure builder to create or edit your custom data structures.

It will automatically select how to version up your data structure depending on the changes you have just made.

In this example, a new required property has been added to the data structure. This is a breaking change, so the builder will increment the first digit:

![Data structure builder showing the \&quot;add\_to\_cart\&quot; event schema with properties user\_id, product\_id, quantity, and a newly added required property \&quot;category\&quot; highlighted in yellow. A banner at the bottom reads \&quot;Version increased to 2-0-0 — your current version is 1-0-0, based on your changes the new version will be 2-0-0.\&quot;](/assets/images/data-structures-2-1133e290be8b44d1ed1d0f74dcef1c3a.png)

In this example, an additional enum option has been added to `category`. This is a non-breaking change, so the builder is incrementing the middle digit:

![Data structure builder Properties section showing the \&quot;category\&quot; field highlighted in yellow with an updated enum list — books, technology, games, and newly added \&quot;appliances\&quot;. A banner at the bottom reads \&quot;Version increased to 2-1-0 — your current version is 2-0-0, based on your changes the new version will be 2-1-0.\&quot;](/assets/images/data-structures-1-984215ae8aba92b496bfca0378c9e0be.png)

## Versioning with the JSON editor

When using the JSON editor, at the point of publishing a data structure you'll be asked to select which version you'd like to create.

![Publish version to development environment dialog for \&quot;subscription\_payment\_failed\&quot; showing four version options: Non-breaking (1-0-1), Non-breaking (1-1-0), Breaking (2-0-0) — currently selected with a recommended upgrade warning — and Patch (1-0-0). A caution banner warns about the impact of selecting the wrong version increment.](/assets/images/json_editor_version_options-bf9596ad2af60a2639de1ee5107f87cd.png)

## Patch a schema

To [patch a schema](/docs/fundamentals/schemas/versioning/#overwrite-a-schema), i.e. apply changes to it without updating the version, select the **Patch** option when saving the schema.

Note that various pipeline components, most importantly Enrich (including Enrich embedded in Snowplow Mini and Snowplow Micro), cache schemas to improve performance. The default caching time is 10 minutes (it's controlled by the [Iglu Resolver configuration](/docs/api-reference/iglu/iglu-resolver/)). This means that the effect of patching a schema will not be immediate.

> **Note:** If you are using Snowplow Self-Hosted, to patch a schema, don't increment the schema version when [uploading it with `igluctl`](/docs/api-reference/iglu/manage-schemas/).
>
> You'll need to explicitly enable patching in the [Iglu Server configuration](/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/) (`patchesAllowed`) at your own risk.

## Mark a schema as superseded

To [mark a schema as superseded](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded), use the JSON editor and add a `$supersedes` field.

---

# Discover and manage events with the Event Catalog
> Browse, search, and manage all event specifications across tracking plans from a single location to improve event discoverability and governance.
> Source: https://docs.snowplow.io/docs/event-studio/event-catalog/

The Event Catalog provides a centralized location to discover and manage all event specifications across your [tracking plans](/docs/event-studio/tracking-plans/). Instead of navigating through individual tracking plans to find specific events, you can browse, search, and filter all event specifications from a single view.

When your organization has multiple tracking plans across different teams and domains, finding specific event specifications can become challenging. The Event Catalog addresses this by:

- **Centralizing discovery**: browse all event specifications in one place rather than searching through individual tracking plans
- **Improving governance**: maintain oversight of all event specifications and their status across your organization
- **Streamlining onboarding**: help new team members understand what events are available and how they're organized
- **Enabling cross-team collaboration**: see how different teams have defined similar events and share best practices

## Access the Event Catalog

Navigate to **Event Catalog** in the main navigation.

![Event Catalog page listing six event specifications across multiple tracking plans, showing their entities, event volume, last seen date, and status (Draft or Published), with a \&quot;Create event specification\&quot; button in the top right](/assets/images/event-catalog-overview-0239dfa65e17f58839465c6c9b30533a.png)

## Browse event specifications

The Event Catalog provides a comprehensive list of all event specifications defined across your tracking plans. Each row displays:

| Column                   | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| Event specification name | The name and schema identifier of the event specification          |
| Entities                 | The [entities](/docs/fundamentals/entities/) attached to the event |
| Tracking plan            | The tracking plan containing the event specification               |
| Volume                   | The number of events collected                                     |
| Last seen                | When the event was last received                                   |
| Status                   | The status of the event specification                              |

### Filter and search

You can filter and search the list to find specific event specifications. Use the controls at the top of the list:

- **Search**: enter text to filter by event specification name
- **Status filter**: show all specifications or filter by Draft or Published status
- **Entity filter**: filter specifications by attached entities

## Create event specifications

From the Event Catalog, you can create new event specifications without first navigating to a tracking plan.

To create an event specification:

1. Click **Create event specification**
2. Enter a name that describes the event
3. Optionally add a description
4. Select the tracking plan this event specification belongs to
5. Select the source applications where this event is tracked
6. Click **Confirm**

![\&quot;Create event specification\&quot; dialog with name \&quot;Add to cart\&quot; entered, tracking plan set to E-commerce Web, and source application set to website](/assets/images/create-event-specification-7ba523f45dd227ae96a8434ee0d0ee4d.png)

After creation, you can add event and entity data structures, triggers, and property instructions.

> **Tip:** Use descriptive names that reflect the business action being tracked. For example, "Add to cart" is clearer than "cart\_event\_1".

---

# Client-side schema and event specification validation
> Enable real-time schema and event specification validation in the browser for JavaScript and TypeScript trackers to catch tracking errors before events are sent, using Snowtype.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/client-side-validation/

Using Snowtype you can get notified, at runtime, if tracked events don't match their schema definitions, or the rules defined in their event specification. This allows you to catch errors before they reach production.

When enabled, every event sent through Snowtype generated code is checked at runtime against:

- **Schema rules**: does the event or entity data match the schema definition?
- **Cardinality rules**: does the event include the correct number of each entity, as defined in the event specification?
- **Property rules**: do entity values match the constraints defined in the event specification?

> **Info:** This feature is available for the [Browser tracker](/docs/sources/web-trackers/quick-start-guide/?platform=browser) only, in both JavaScript and TypeScript.

## Generate code with validations

To use validation, you'll first need to install three [Ajv](https://ajv.js.org/) packages: `ajv@8`, `ajv-formats@2` and `ajv-draft-04@1`.

```bash
# Example using npm

npm install ajv@8 ajv-formats@2 ajv-draft-04@1
```

To opt-in to client-side validation, include the `--validations` flag when you are generating your code.

```bash
npx snowtype generate --validations
```

You can also enable validations permanently in your [configuration file](/docs/event-studio/implement-tracking/configuration-reference/) instead of passing the flag each time.

Snowtype will generate:

- A `validations.ts` file that contains the schema details for all the listed sources
- Additional helper functions
- Additional processing for each `track` function
- Additional processing for each `trackXSpec` function

All the changes are under the hood, and don't affect how you call the generated functions. Continue to track events as normal, and any validation errors will appear in the browser console.

### Updated `track` functions

To validate schemas, Snowtype adds extra code to the [generated `track` functions](/docs/event-studio/implement-tracking/generate-tracking-code/#data-structures).

The highlighted lines in this example show the changed or additional code:

```typescript
// Ajv helper functions omitted for brevity



/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 */

export function trackWebPage<T extends {} = any>(webPage: WebPage & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, __esId, ...data } = webPage as typeof webPage & EventSpecIdInput;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data

    };

    [event].concat(context || []).forEach(s => validateSchema(s, __esId));

    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}
```

### Updated `trackXSpec` functions

To validate event specification instructions and cardinality rules, Snowtype adds extra code to the [generated `trackXSpec` functions](/docs/event-studio/implement-tracking/generate-tracking-code/#event-specifications).

The highlighted lines in this example show the changed or additional code:

```typescript
// Ajv helper functions omitted for brevity



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 */

export function trackUserLogInSpec(userLogIn: Login & ContextsOrTimestamp<User | UserAuthentication>, trackers?: string[]){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow - Example',

        data_product_domain: 'Marketing'

    });

    validateEntityRules(

        'a965caf1-88a6-4a89-9aea-cc92516a9d56' as keyof typeof entityRules,

        userLogIn.context as unknown as SelfDescribingJson[]

    );

    const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        context,

        /* Used for runtime validation only */

        __esId: 'a965caf1-88a6-4a89-9aea-cc92516a9d56'

    };



    trackLogin<User | UserAuthentication | EventSpecification>(modifiedUserLogIn, trackers);

}
```

### Bundle size consideration

Because of the additional libraries required for client-side validation, using validation will increase your application bundle size.

For this reason, we recommend using validation only in development and test environments.

## Schema validation

When you call a generated `trackX` function, it will validate the event data against the schema. It will also validate any attached entities, as long as Snowtype has a reference of their schemas in the `validations.ts` file.

For JavaScript projects, this type of validation is especially useful given the lack of types. For TypeScript, it's an additional layer of protection that can also catch mistakes in dynamic or loosely typed values.

Your app will raise a `Snowtype Schema Validation error` warning in the browser console if a property violates the schema. In this example, the property `id` is expected to be a string, but the value `1` is passed instead. The erroneous value can be found under `errors[n].data`, and a stack trace pointing to the calling function is also provided:

![Browser console showing a \&quot;Snowtype Schema Validation error\&quot; warning where the id property received 1 (a number) instead of the expected string type, with the invalid value shown under errors\[n\].data and a stack trace pointing to the calling function](/assets/images/validation-0a7cf374318205444a962c9a69fbc40d.png)

## Entity cardinality rule validation

[Cardinality rules](/docs/event-studio/tracking-plans/) define how many of a given entity an event specification expects. For example:

- Exactly 1
- At least 1
- Between 1 and 2

With client-side validation enabled, your Snowtype code will check these rules at runtime. If the event includes the wrong amount of a required entity, a `Snowtype Entity Cardinality Rules error` warning appears in the browser console.

For example, with a `product` entity that has a cardinality rule of "Exactly 1":

![Event specification in Snowplow Console showing a product entity with a cardinality rule set to \&quot;Exactly 1\&quot;](/assets/images/cardinality-validation-7387c2d3b0d509881ac5973741cce55a.png)

This code satisfies the rule:

```typescript
trackButtonClickSpec({

    label: "Product click",

    context: [createProduct({ name: "Product", price: 1, quantity: 1 })],

});
```

Adding a second `product` entity violates the rule:

```typescript
trackButtonClickSpec({

    label: "Product click",

    context: [

        createProduct({ name: "Product", price: 1, quantity: 1 }),

        // This violates the cardinality rule of Exactly 1

        createProduct({ name: "Product 2", price: 1, quantity: 1 }),

    ],

});
```

![Browser console showing a \&quot;Snowtype Entity Cardinality Rules error\&quot; warning with minCardinality, maxCardinality, and currentCardinality values listed, and a stack trace pointing to the calling function](/assets/images/cardinality-browser-6e2ea31e9b212020037906f5da74e21d.png)

The warning includes the `minCardinality` and `maxCardinality` expected, the `currentCardinality` i.e. how many were actually passed, and a stack trace.

A warning also appears when a required entity is missing entirely:

![Browser console showing a \&quot;Snowtype Entity Cardinality Rules error\&quot; warning for a required entity that was not included in the tracked event](/assets/images/cardinality-empty-fed6ca67310fc3b06dd24ea53a735b26.png)

## Property instructions validation

[Property rule](/docs/event-studio/tracking-plans/create-and-manage/) instructions are constraints you define on event or entity properties within an event specification.

Snowtype creates [types for property rules](/docs/event-studio/implement-tracking/generate-tracking-code/#property-rules-and-instructions) for all trackers. This additional runtime validation layer can catch any mistakes that slip through, such as when using dynamic values.

In this example event specification, the `category` property of the `product` entity must be either `"related"` or `"cross-sell"`.

![Event specification in Snowplow Console showing the category property of a product entity with allowed values restricted to \&quot;related\&quot; and \&quot;cross-sell\&quot;](/assets/images/property-rules-browser-0895b2997955f456740a1bf4db235428.png)

This code satisfies the property rule:

```typescript
trackRelatedSpec({

    label: "Related product",

    context: [

        createProductRelated({

            category: "cross-sell",

            name: "product",

            quantity: 1,

            price: 10,

        }),

    ],

});
```

Passing a value outside the allowed set triggers a `Snowtype Schema Validation error` warning:

```typescript
trackRelatedSpec({

    label: "Related product",

    context: [

        createProductRelated({

            // "cross-sells" has a typo and is not a valid category

            category: "cross-sells",

            name: "product",

            quantity: 1,

            price: 10,

        }),

    ],

});
```

![Browser console showing a \&quot;Snowtype Schema Validation error\&quot; warning triggered by passing \&quot;cross-sells\&quot; for a category property that only accepts \&quot;related\&quot; or \&quot;cross-sell\&quot;](/assets/images/property-rules-enum-error-1efab496e8e3d763cad18c269380dc1b.png)

## Custom violation handlers

By default, violations are logged to the browser console using `console.warn`. You can override this behavior with a custom `violationsHandler` using the `snowtype.setOptions` API:

```typescript
import { snowtype } from "{{outpath}}/snowplow";



function myViolationsHandler(error) {

    // Custom violation handling logic

}



snowtype.setOptions({ violationsHandler: myViolationsHandler });
```

The `error` argument has the following shape:

```typescript
type ErrorType = {

    /** Error code number (e.g. 100, 200, 201) */

    code: number;

    /** Error message */

    message: string;

    /** Description of the violation */

    description: string;

    /** Details of the violations */

    errors: (ErrorObject | Record<string, unknown>)[];

};
```

This is useful for:

- Unit testing: throw an `Error` so that tests fail automatically when a violation occurs
- Error monitoring: report violations to a service like Sentry in staging or production

> **Tip:** When `NODE_ENV` is set to `test`, as many testing libraries do automatically, the default handler throws an `Error` instead of logging a warning. This means tracking violations will cause your tests to fail without any extra configuration.

## Pipeline validation and `pattern`

When you run `snowtype generate`, Snowtype will print a warning if it detects [`pattern` (regex)](https://json-schema.org/understanding-json-schema/reference/string#regexp) properties in any of your schemas:

```bash
✔: Code generated, writing...

⚠ Warning: Some schemas appear to have attributes that can have inconsistent validations between the pipeline and client-side validation.
```

The Snowplow pipeline uses Java/Scala regex, while client-side validation uses JavaScript regex. This can lead to differences in validation results, especially for complex regex patterns. In most cases, you can ignore this warning.

---

# Snowtype CLI command reference
> Summary of Snowtype CLI commands and options for code generation, with details on usage scenarios and configuration.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/command-reference/

This page lists every Snowtype CLI command and its options. For more details about configuring Snowtype, see the [sources and output configuration](/docs/event-studio/implement-tracking/snowtype-config/) and [configuration reference](/docs/event-studio/implement-tracking/configuration-reference/) pages.

Use each command like this:

```bash
snowtype [COMMAND] [OPTIONS] [CONTEXT-SPECIFIC-OPTIONS]



# Example:

snowtype init -i my-organization-id -t snowplow-android-tracker -l kotlin -o ./generated-code
```

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

## Global options

These options work with any command.

| Flag                      | Description                        |
| ------------------------- | ---------------------------------- |
| `-h, --help`              | Show help for the command.         |
| `-V, --version`           | Output the package version number. |
| `-k, --apiKey <string>`   | Snowplow Console API key.          |
| `-s, --apiKeyId <string>` | Snowplow Console API key ID.       |
| `-v, --verbose`           | Enable verbose logging.            |

## `init`

Initializes Snowtype in a project by creating the configuration file.

| Flag                            | Description                    |
| ------------------------------- | ------------------------------ |
| `-i, --organizationId <string>` | Organization ID                |
| `-t, --tracker <tracker>`       | Tracker to use                 |
| `-l, --language <language>`     | Language to use                |
| `-o, --outpath <string>`        | Output path for generated code |

See the [configuration reference](/docs/event-studio/implement-tracking/configuration-reference/#supported-trackers-and-languages) for details of the valid tracker and language options.

## `generate`

Generates tracking code based on the configuration file.

| Flag                                    | Description                                                                                                                                                                              | Default |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `-c, --config <string>`                 | Config file path.                                                                                                                                                                        |         |
| `--instructions`                        | Generate [event specification instructions](/docs/event-studio/implement-tracking/generate-tracking-code/#markdown-instructions).                                                        |         |
| `--no-instructions`                     | Generate without instructions.                                                                                                                                                           |         |
| `--validations`                         | Add [runtime validation](/docs/event-studio/implement-tracking/client-side-validation/) using Ajv. Available only for the Browser tracker.                                               |         |
| `--no-validations`                      | Generate without runtime validations.                                                                                                                                                    |         |
| `--disallowDevSchemas`                  | Disallow [generation of code](/docs/event-studio/implement-tracking/generate-tracking-code/#prevent-generation-from-development-schemas) using schemas only deployed on DEV environment. | `false` |
| `--deprecateOnlyOnProdAvailableUpdates` | Show [deprecation warnings](/docs/event-studio/implement-tracking/generate-tracking-code/#deprecation-and-hidden-warnings) only when there are PROD-available schema updates.            | `false` |

## `update`

Checks for updates to event specifications and data structures. Event specification updates are checked against the versions pinned in `.snowtype-lock.json`. Data structure updates are checked against versions in `snowtype.config.json`.

| Flag                          | Description                                                                                                                                                                                       | Default |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `-c, --config <string>`       | Config file path.                                                                                                                                                                                 |         |
| `-y, --yes`                   | Apply all available updates without prompting. Overwrites the configuration and lock files.                                                                                                       | `false` |
| `-m, --maximumBump <level>`   | For data structures and schemas only. Maximum [SchemaVer](/docs/fundamentals/schemas/versioning/#schema-version-format) bump to show update notifications for. Values: `patch`, `minor`, `major`. | `major` |
| `-d, --latestDraft`           | Include draft versions when checking for event specification updates. By default, only published versions are shown. Does not affect data structures.                                             |         |
| `-e, --eventSpecs <ids...>`   | Check only the specified event specifications.                                                                                                                                                    |         |
| `-p, --dataProducts <ids...>` | Check only event specifications belonging to the specified tracking plans.                                                                                                                        |         |

## `patch`

Adds new items to the configuration file without modifying it by hand.

| Flag                                   | Description                                                                                |
| -------------------------------------- | ------------------------------------------------------------------------------------------ |
| `-c, --config <string>`                | Config file path                                                                           |
| `-e, --eventSpecificationIds <ids...>` | Event specification IDs                                                                    |
| `-p, --dataProductIds <ids...>`        | Tracking plan IDs                                                                          |
| `-d, --dataStructures <uris...>`       | Data structure schema URIs                                                                 |
| `-i, --igluCentralSchemas <uris...>`   | Iglu Central schema URI.                                                                   |
| `-r, --repositories <paths...>`        | [Local data structure repository](/docs/api-reference/snowplow-cli/data-structures/) paths |

## `purge`

Removes event specification entries from the lock file that no longer appear in the configuration file.

| Flag                    | Description      |
| ----------------------- | ---------------- |
| `-c, --config <string>` | Config file path |

---

# Configuration reference
> Complete reference for all Snowtype configuration file options, including schema sources, output settings, and command behavior.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/configuration-reference/

Snowtype reads its settings from a configuration file in your project root. The file is created when you run [`snowtype init`](/docs/event-studio/implement-tracking/install-snowtype/), and you can edit it directly at any time.

Keep the configuration file in the root of your project, and include it in your version control system.

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

## Configuration file structure

The configuration file is a JSON, JavaScript, or TypeScript object. You can choose format which to use during initialization. As JSON, it has the following structure:

```json
{

    "organizationId": "a654321b-c111-33d3-e321-1f123456789g",

    "tracker": "@snowplow/browser-tracker",

    "language": "typescript",

    "outpath": "./src/snowtype",

    "eventSpecificationIds": [

        "a123456b-c222-11d1-e123-1f123456789g"

    ],

    "dataProductIds": [

        "a123456b-c222-11d1-e123-1f12345678dp"

    ],

    "dataStructures": [

        "iglu:com.myorg/custom_web_page/jsonschema/1-1-0"

    ],

    "igluCentralSchemas": [

        "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

    ],

    "repositories": ["../data-structures"],

    "namespace": "Snowtype",

    "options": {

        "commands": {

            "generate": {

                "instructions": true,

                "validations": true,

                "disallowDevSchemas": true

            },

            "update": {

                "maximumBump": "minor",

                "showOnlyProdUpdates": true

            },

            "patch": {

                "regenerateOnPatch": false

            }

        }

    }

}
```

| Property                | Type       | Description                                                                                                                                                                                      | Required |
| ----------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `organizationId`        | `string`   | Your Snowplow account organization ID. Used to fetch tracking plans, event specifications, or data structures from Console. Not required if you're only generating code from a local repository. | ✅/❌      |
| `tracker`               | `string`   | The Snowplow tracker to generate code for. This determines the structure of the generated code and which tracker it will work with.                                                              | ✅        |
| `language`              | `string`   | The programming language to generate code in.                                                                                                                                                    | ✅        |
| `outpath`               | `string`   | The output path where Snowtype writes the generated code, relative to the current working directory when running the script.                                                                     | ✅        |
| `dataProductIds`        | `string[]` | An array of [tracking plan](/docs/event-studio/tracking-plans/) IDs to generate code for.                                                                                                        | ✅        |
| `eventSpecificationIds` | `string[]` | An array of [event specification](/docs/event-studio/tracking-plans/event-specifications/) IDs to generate code for.                                                                             | ✅        |
| `dataStructures`        | `string[]` | An array of schema URIs for [data structures](/docs/fundamentals/schemas/) published in Console to generate code for.                                                                            | ✅        |
| `igluCentralSchemas`    | `string[]` | An array of schema URIs for [Iglu Central](https://iglucentral.com/) schemas to generate code for.                                                                                               | ❌        |
| `repositories`          | `string[]` | An array of paths to local [Snowplow CLI](/docs/api-reference/snowplow-cli/data-structures/) schema repositories to generate code for.                                                           | ❌        |
| `namespace`             | `string`   | Swift tracker only: a namespace prefix for the generated code. For other trackers it will trigger an error.                                                                                      | ❌        |
| `options`               | `object`   | Additional settings related to Snowtype behavior. See details below.                                                                                                                             | ❌        |

To find your organization ID, go to [Console](https://console.snowplowanalytics.com/settings) > **Settings** > **Manage organization**.

### File formats

Most of the examples in this documentation show the configuration file in JSON format, but you can also use JavaScript or TypeScript if you prefer. The filename must start with `snowtype.config` — for example, `snowtype.config.json`, `snowtype.config.js`, or `snowtype.config.ts`.

**JavaScript:**

```javascript
const config = {

    "organizationId": "a654321b-c111-33d3-e321-1f123456789g",

    "tracker": "@snowplow/browser-tracker",

    "language": "typescript",

    "outpath": "./src/snowtype",

    "eventSpecificationIds": [

        "a123456b-c222-11d1-e123-1f123456789g"

    ],

    "dataProductIds": [

        "a123456b-c222-11d1-e123-1f12345678dp"

    ],

    "dataStructures": [

        "iglu:com.myorg/custom_web_page/jsonschema/1-1-0"

    ],

    "igluCentralSchemas": [

        "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

    ],

    "repositories": ["../data-structures"],

    "namespace": "Snowtype",

    "options": {

        "commands": {

            "generate": {

                "instructions": true,

                "validations": true,

                "disallowDevSchemas": true

            },

            "update": {

                "maximumBump": "minor",

                "showOnlyProdUpdates": true

            },

            "patch": {

                "regenerateOnPatch": false

            }

        }

    }

};



module.exports = config;
```

**TypeScript:**

```typescript
const config = {

    organizationId: "a654321b-c111-33d3-e321-1f123456789g",

    tracker: "@snowplow/browser-tracker" as const,

    language: "typescript" as const,

    outpath: "./src/snowtype",

    eventSpecificationIds: [

        "a123456b-c222-11d1-e123-1f123456789g"

    ],

    dataProductIds: [

        "a123456b-c222-11d1-e123-1f12345678dp"

    ],

    dataStructures: [

        "iglu:com.myorg/custom_web_page/jsonschema/1-1-0"

    ],

    igluCentralSchemas: [

        "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

    ],

    repositories: ["../data-structures"],

    namespace: "Snowtype",

    options: {

        commands: {

            generate: {

                instructions: true,

                validations: true,

                disallowDevSchemas: true

            },

            update: {

                maximumBump: "minor",

                showOnlyProdUpdates: true

            },

            patch: {

                regenerateOnPatch: false

            }

        }

    }

};



export default config;
```

***

## Supported trackers and languages

Set `tracker` and `language` to a valid combination from this table:

| Tracker                                                 | `tracker`                        | `language`                 |
| ------------------------------------------------------- | -------------------------------- | -------------------------- |
| [Browser](/docs/sources/web-trackers/)                  | `@snowplow/browser-tracker`      | `javascript`, `typescript` |
| [JavaScript](/docs/sources/web-trackers/)               | `@snowplow/javascript-tracker`   | `javascript`               |
| [iOS](/docs/sources/mobile-trackers/)                   | `snowplow-ios-tracker`           | `swift`                    |
| [Android](/docs/sources/mobile-trackers/)               | `snowplow-android-tracker`       | `kotlin`                   |
| [React Native](/docs/sources/react-native-tracker/)     | `@snowplow/react-native-tracker` | `typescript`               |
| [Flutter](/docs/sources/flutter-tracker/)               | `snowplow-flutter-tracker`       | `dart`                     |
| [Node.js](/docs/sources/node-js-tracker/)               | `@snowplow/node-tracker`         | `javascript`, `typescript` |
| [Go](/docs/sources/golang-tracker/)                     | `snowplow-golang-tracker`        | `go`                       |
| [Java](/docs/sources/java-tracker/)                     | `snowplow-java-tracker`          | `java`                     |
| [Google Tag Manager](/docs/sources/google-tag-manager/) | `google-tag-manager`             | `javascript-gtm`           |

## Command options

The `options` object configures the behavior of specific Snowtype commands. You can set these in the configuration file so they apply on every run, or override them with CLI flags. See the [commands reference](/docs/event-studio/implement-tracking/command-reference/) for more details on the flags.

> **Note:** CLI flags take precedence over values set in the configuration file.

### `generate`

The `generate` options apply when running `snowtype generate`, which creates the initial code based on your configuration and schemas.

| Option                                | Type      | Default | Description                                                                                                                                                                                  | Equivalent flag                         |
| ------------------------------------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| `instructions`                        | `boolean` | `false` | Generate a [Markdown file](/docs/event-studio/implement-tracking/generate-tracking-code/#markdown-instructions) with implementation instructions for each event specification                | `--instructions`                        |
| `validations`                         | `boolean` | `false` | Include runtime [client-side validation](/docs/event-studio/implement-tracking/client-side-validation/) code in the output                                                                   | `--validations`                         |
| `disallowDevSchemas`                  | `boolean` | `false` | Fail generation if any schema is only deployed to the development environment                                                                                                                | `--disallowDevSchemas`                  |
| `deprecateOnlyOnProdAvailableUpdates` | `boolean` | `false` | Only show [deprecation warnings](/docs/event-studio/implement-tracking/generate-tracking-code/#deprecation-and-hidden-warnings) for schemas that have a newer version deployed to production | `--deprecateOnlyOnProdAvailableUpdates` |

```json
{

    "options": {

        "commands": {

            "generate": {

                "instructions": true,

                "validations": true,

                "disallowDevSchemas": true

            }

        }

    }

}
```

### `update`

The `update` options apply when running `snowtype update`, which checks for new schema versions and regenerates code if needed.

| Option                | Type                                | Default   | Description                                                                                                                                                      |
| --------------------- | ----------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `regenerateOnUpdate`  | `boolean`                           | `true`    | Automatically regenerate code after accepting updates.                                                                                                           |
| `maximumBump`         | `"major"` \| `"minor"` \| `"patch"` | `"major"` | The highest [SchemaVer](/docs/fundamentals/schemas/) bump level to include in update checks. For example, setting this to `"minor"` hides major version updates. |
| `showOnlyProdUpdates` | `boolean`                           | `false`   | Only show updates for data structures that have been deployed to the production environment.                                                                     |

```json
{

    "options": {

        "commands": {

            "update": {

                "maximumBump": "minor",

                "showOnlyProdUpdates": true

            }

        }

    }

}
```

### `patch`

The `patch` option applies when running `snowtype patch`, which adds new sources to your configuration file and optionally regenerates code.

| Option              | Type      | Default | Description                                                                   |
| ------------------- | --------- | ------- | ----------------------------------------------------------------------------- |
| `regenerateOnPatch` | `boolean` | `true`  | Automatically regenerate code after adding new schemas with `snowtype patch`. |

```json
{

    "options": {

        "commands": {

            "patch": {

                "regenerateOnPatch": false

            }

        }

    }

}
```

## Swift namespace

The `namespace` option applies only when generating Swift code. It adds a prefix to all generated classes, which helps avoid naming conflicts in your project.

For example, setting `namespace` to `"Snowtype"`:

```json
{

    "tracker": "snowplow-ios-tracker",

    "language": "swift",

    "outpath": "./Sources/Tracking",

    "namespace": "Snowtype"

}
```

This results in classes accessed with the `Snowtype` prefix:

```swift
let data = Snowtype.AccountConfirmed(

    companyCountry: "US",

    companyName: "Acme Corp"

)
```

Setting a `namespace` when generating code for a non-Swift tracker will trigger an error.

## Full configuration type

For TypeScript users, this is the complete type definition for the configuration file:

```typescript
type SnowtypeConfig = {

    tracker:

        | "@snowplow/browser-tracker"

        | "@snowplow/javascript-tracker"

        | "@snowplow/node-tracker"

        | "@snowplow/react-native-tracker"

        | "snowplow-android-tracker"

        | "snowplow-ios-tracker"

        | "snowplow-golang-tracker"

        | "snowplow-flutter-tracker"

        | "snowplow-java-tracker";

    language:

        | "typescript"

        | "javascript"

        | "kotlin"

        | "swift"

        | "go"

        | "dart"

        | "java";

    outpath: string;

    organizationId?: string;

    eventSpecificationIds?: string[];

    dataProductIds?: string[];

    dataStructures?: string[];

    igluCentralSchemas?: string[];

    repositories?: string[];

    namespace?: string;

    options?: {

        commands: {

            generate?: {

                instructions?: boolean;

                validations?: boolean;

                disallowDevSchemas?: boolean;

                deprecateOnlyOnProdAvailableUpdates?: boolean;

            };

            update?: {

                regenerateOnUpdate?: boolean;

                maximumBump?: "major" | "minor" | "patch";

                showOnlyProdUpdates?: boolean;

            };

            patch?: {

                regenerateOnPatch?: boolean;

            };

        };

    };

};
```

---

# Example Snowtype generated code for data structures
> Example configuration files, generated output, and how to use them for each Snowtype-supported tracker and language.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/generate-tracking-code/example-data-structures/

The following examples show the Snowtype [configuration file](/docs/event-studio/implement-tracking/snowtype-config/), generated output, and how to use them, for the [Iglu Central](https://iglucentral.com) schema `web_page` and a custom data structure, `product`. The `tracker` and `language` fields in your configuration determine the language and structure of the output.

These are the example schemas:

### `web_page`

**Type:** Schema

Schema for a web page

**Properties:**

| Property      | Description |
| ------------- | ----------- |
| `id` _string_ | _Required._ |

### `product`

**Type:** Schema

Schema for a product entity

**Properties:**

| Property            | Description                                        |
| ------------------- | -------------------------------------------------- |
| `id` _string_       | _Required._ The product ID (SKU).                  |
| `name` _string_     | _Required._ The product name.                      |
| `currency` _string_ | _Required._ The currency the product is listed in. |
| `price` _number_    | _Required._ The price of the product.              |
| `category` _string_ | _Required._ The product category.                  |

## Web trackers

Snowtype can generate code in TypeScript or JavaScript, depending which version of the [web trackers](/docs/sources/web-trackers/) you're using.

### Browser (TypeScript)

Generated code for the [Browser tracker](/docs/sources/web-trackers/). Snowtype produces TypeScript type definitions with `track` and `create` functions that wrap `trackSelfDescribingEvent`.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```typescript
import { trackSelfDescribingEvent, CommonEventProperties, SelfDescribingJson } from '@snowplow/browser-tracker';

// Automatically generated by Snowtype



/**

 * Schema for a web page

 */

export type WebPage = {

    /**

     * The page view ID.

     */

    id: string;

}



/**

 * Schema for a product entity

 */

export type Product = {

    /**

     * The product category.

     */

    category: string;

    /**

     * The currency the product is listed in.

     */

    currency: string;

    /**

     * The product ID (SKU).

     */

    id: string;

    /**

     * The product name.

     */

    name: string;

    /**

     * The price of the product.

     */

    price: number;

}



type ContextsOrTimestamp<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribingJson<T>[] | null | undefined }

/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 */

export function trackWebPage<T extends {} = any>(webPage: WebPage & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = webPage;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow WebPage entity.

 */

export function createWebPage(webPage: WebPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 */

export function trackProduct<T extends {} = any>(product: Product & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = product;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow Product entity.

 */

export function createProduct(product: Product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

Here's how you could use the generated code:

```typescript
import {

  trackWebPage,

  createProduct,

  WebPage,

  Product,

  createWebPage,

} from "./src/tracking/snowplow";



/* Track a self-describing event */

trackWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const product = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product],

});



/* Enforce specific entity types using type arguments */

const webPage = createWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

});

trackWebPage<Product | WebPage>({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product, webPage],

});
```

### Browser (JavaScript)

Generated code for the [Browser tracker](/docs/sources/web-trackers/) in JavaScript. Snowtype produces JSDoc typedefs, with `track` and `create` functions.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "javascript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```javascript
import { trackSelfDescribingEvent } from '@snowplow/browser-tracker';

// Automatically generated by Snowtype



/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Schema for a web page

 * @typedef {object} WebPage

 * @property {string} id The page view ID.

 */





/**

 * Typedef for a WebPage entity context.

 * @typedef {object} WebPageContext

 * @property {string} schema The schema of the context.

 * @property { WebPage } data The data to send for the context.

 */



/**

 * Schema for a product entity

 * @typedef {object} Product

 * @property {string} category The product category.

 * @property {string} currency The currency the product is listed in.

 * @property {string} id The product ID (SKU).

 * @property {string} name The product name.

 * @property {number} price The price of the product.

 */





/**

 * Typedef for a Product entity context.

 * @typedef {object} ProductContext

 * @property {string} schema The schema of the context.

 * @property { Product } data The data to send for the context.

 */



/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 * @param { WebPage & Timestamp & Context } webPage - Attributes for WebPage.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

export function trackWebPage(webPage, trackers){

    const { context, timestamp, ...data } = webPage;

    const event = {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow WebPage entity.

 * @param { WebPage } webPage - Attributes for WebPage.

 */

export function createWebPage(webPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 * @param { Product & Timestamp & Context } product - Attributes for Product.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

export function trackProduct(product, trackers){

    const { context, timestamp, ...data } = product;

    const event = {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow Product entity.

 * @param { Product } product - Attributes for Product.

 */

export function createProduct(product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

Here's how you could use the generated code:

```javascript
import {

  trackWebPage,

  createProduct,

  createWebPage,

} from "./src/tracking/snowplow";



/* Track a self-describing event */

trackWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const product = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product],

});
```

### JavaScript tag

Generated code for the [JavaScript tag](/docs/sources/web-trackers/). Snowtype produces functions that call `window.snowplow()` directly, suitable for use with the Snowplow JavaScript tag loaded via a script element.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/javascript-tracker",

  "language": "javascript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```javascript
// Automatically generated by Snowtype



/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Schema for a web page

 * @typedef {object} WebPage

 * @property {string} id The page view ID.

 */





/**

 * Typedef for a WebPage entity context.

 * @typedef {object} WebPageContext

 * @property {string} schema The schema of the context.

 * @property { WebPage } data The data to send for the context.

 */



/**

 * Schema for a product entity

 * @typedef {object} Product

 * @property {string} category The product category.

 * @property {string} currency The currency the product is listed in.

 * @property {string} id The product ID (SKU).

 * @property {string} name The product name.

 * @property {number} price The price of the product.

 */





/**

 * Typedef for a Product entity context.

 * @typedef {object} ProductContext

 * @property {string} schema The schema of the context.

 * @property { Product } data The data to send for the context.

 */



/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 * @param { WebPage & Timestamp & Context } webPage - Attributes for WebPage.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

function trackWebPage(webPage, trackers){

    const trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

    const { context, timestamp, ...data } = webPage;

    window.snowplow('trackSelfDescribingEvent' + trackerNames, {

        event: {

            schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

            data

        },

        context,

        timestamp,

    });

}



/**

 * Creates a Snowplow WebPage entity.

 * @param { WebPage } webPage - Attributes for WebPage.

 */

function createWebPage(webPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 * @param { Product & Timestamp & Context } product - Attributes for Product.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

function trackProduct(product, trackers){

    const trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

    const { context, timestamp, ...data } = product;

    window.snowplow('trackSelfDescribingEvent' + trackerNames, {

        event: {

            schema: 'iglu:com.example/product/jsonschema/1-0-0',

            data

        },

        context,

        timestamp,

    });

}



/**

 * Creates a Snowplow Product entity.

 * @param { Product } product - Attributes for Product.

 */

function createProduct(product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

Here's how you could use the generated code:

```javascript
/* Track a self-describing event */

trackWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const product = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product],

});
```

## iOS (Swift)

Generated code for the [iOS tracker](/docs/sources/mobile-trackers/). Snowtype produces Swift structs with `toEvent()` and `toEntity()` methods.

The generated code is compatible with both Swift 5.5+ and 6.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-ios-tracker",

  "language": "swift",

  "outpath": "./Sources/Tracking/Snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ],

  // "namespace": "Snowtype"

}
```

When generating tracking code for the iOS tracker, you can set the optional `namespace` option in your Snowtype configuration. All classes generated will be included in this namespace. For example, instead of generating `WebPage` and `Product` structs as in the example, Snowtype would generate `Snowtype.WebPage` and `Snowtype.Product`. This can be helpful for avoiding naming conflicts with other types in your codebase.

````swift
import Foundation

import SnowplowTracker



// MARK: - Encode/decode helpers



final class JSONNull: Codable, Hashable, Sendable {



        public static func == (lhs: JSONNull, rhs: JSONNull) -> Bool {

                        return true

        }



        public func hash(into hasher: inout Hasher) {}



        public init() {}



        public required init(from decoder: Decoder) throws {

                        let container = try decoder.singleValueContainer()

                        if !container.decodeNil() {

                                        throw DecodingError.typeMismatch(JSONNull.self, DecodingError.Context(codingPath: decoder.codingPath, debugDescription: "Wrong type for JSONNull"))

                        }

        }



        public func encode(to encoder: Encoder) throws {

                        var container = encoder.singleValueContainer()

                        try container.encodeNil()

        }

}



final class JSONCodingKey: CodingKey, Sendable {

        let key: String



        required init?(intValue: Int) {

                        return nil

        }



        required init?(stringValue: String) {

                        key = stringValue

        }



        var intValue: Int? {

                        return nil

        }



        var stringValue: String {

                        return key

        }

}



class JSONAny: Codable, @unchecked Sendable {



        let value: Any



        static func decodingError(forCodingPath codingPath: [CodingKey]) -> DecodingError {

                        let context = DecodingError.Context(codingPath: codingPath, debugDescription: "Cannot decode JSONAny")

                        return DecodingError.typeMismatch(JSONAny.self, context)

        }



        static func encodingError(forValue value: Any, codingPath: [CodingKey]) -> EncodingError {

                        let context = EncodingError.Context(codingPath: codingPath, debugDescription: "Cannot encode JSONAny")

                        return EncodingError.invalidValue(value, context)

        }



        static func decode(from container: SingleValueDecodingContainer) throws -> Any {

                        if let value = try? container.decode(Bool.self) {

                                        return value

                        }

                        if let value = try? container.decode(Int64.self) {

                                        return value

                        }

                        if let value = try? container.decode(Double.self) {

                                        return value

                        }

                        if let value = try? container.decode(String.self) {

                                        return value

                        }

                        if container.decodeNil() {

                                        return JSONNull()

                        }

                        throw decodingError(forCodingPath: container.codingPath)

        }



        static func decode(from container: inout UnkeyedDecodingContainer) throws -> Any {

                        if let value = try? container.decode(Bool.self) {

                                        return value

                        }

                        if let value = try? container.decode(Int64.self) {

                                        return value

                        }

                        if let value = try? container.decode(Double.self) {

                                        return value

                        }

                        if let value = try? container.decode(String.self) {

                                        return value

                        }

                        if let value = try? container.decodeNil() {

                                        if value {

                                                        return JSONNull()

                                        }

                        }

                        if var container = try? container.nestedUnkeyedContainer() {

                                        return try decodeArray(from: &container)

                        }

                        if var container = try? container.nestedContainer(keyedBy: JSONCodingKey.self) {

                                        return try decodeDictionary(from: &container)

                        }

                        throw decodingError(forCodingPath: container.codingPath)

        }



        static func decode(from container: inout KeyedDecodingContainer<JSONCodingKey>, forKey key: JSONCodingKey) throws -> Any {

                        if let value = try? container.decode(Bool.self, forKey: key) {

                                        return value

                        }

                        if let value = try? container.decode(Int64.self, forKey: key) {

                                        return value

                        }

                        if let value = try? container.decode(Double.self, forKey: key) {

                                        return value

                        }

                        if let value = try? container.decode(String.self, forKey: key) {

                                        return value

                        }

                        if let value = try? container.decodeNil(forKey: key) {

                                        if value {

                                                        return JSONNull()

                                        }

                        }

                        if var container = try? container.nestedUnkeyedContainer(forKey: key) {

                                        return try decodeArray(from: &container)

                        }

                        if var container = try? container.nestedContainer(keyedBy: JSONCodingKey.self, forKey: key) {

                                        return try decodeDictionary(from: &container)

                        }

                        throw decodingError(forCodingPath: container.codingPath)

        }



        static func decodeArray(from container: inout UnkeyedDecodingContainer) throws -> [Any] {

                        var arr: [Any] = []

                        while !container.isAtEnd {

                                        let value = try decode(from: &container)

                                        arr.append(value)

                        }

                        return arr

        }



        static func decodeDictionary(from container: inout KeyedDecodingContainer<JSONCodingKey>) throws -> [String: Any] {

                        var dict = [String: Any]()

                        for key in container.allKeys {

                                        let value = try decode(from: &container, forKey: key)

                                        dict[key.stringValue] = value

                        }

                        return dict

        }



        static func encode(to container: inout UnkeyedEncodingContainer, array: [Any]) throws {

                        for value in array {

                                        if let value = value as? Bool {

                                                        try container.encode(value)

                                        } else if let value = value as? Int64 {

                                                        try container.encode(value)

                                        } else if let value = value as? Double {

                                                        try container.encode(value)

                                        } else if let value = value as? String {

                                                        try container.encode(value)

                                        } else if value is JSONNull {

                                                        try container.encodeNil()

                                        } else if let value = value as? [Any] {

                                                        var container = container.nestedUnkeyedContainer()

                                                        try encode(to: &container, array: value)

                                        } else if let value = value as? [String: Any] {

                                                        var container = container.nestedContainer(keyedBy: JSONCodingKey.self)

                                                        try encode(to: &container, dictionary: value)

                                        } else {

                                                        throw encodingError(forValue: value, codingPath: container.codingPath)

                                        }

                        }

        }



        static func encode(to container: inout KeyedEncodingContainer<JSONCodingKey>, dictionary: [String: Any]) throws {

                        for (key, value) in dictionary {

                                        let key = JSONCodingKey(stringValue: key)!

                                        if let value = value as? Bool {

                                                        try container.encode(value, forKey: key)

                                        } else if let value = value as? Int64 {

                                                        try container.encode(value, forKey: key)

                                        } else if let value = value as? Double {

                                                        try container.encode(value, forKey: key)

                                        } else if let value = value as? String {

                                                        try container.encode(value, forKey: key)

                                        } else if value is JSONNull {

                                                        try container.encodeNil(forKey: key)

                                        } else if let value = value as? [Any] {

                                                        var container = container.nestedUnkeyedContainer(forKey: key)

                                                        try encode(to: &container, array: value)

                                        } else if let value = value as? [String: Any] {

                                                        var container = container.nestedContainer(keyedBy: JSONCodingKey.self, forKey: key)

                                                        try encode(to: &container, dictionary: value)

                                        } else {

                                                        throw encodingError(forValue: value, codingPath: container.codingPath)

                                        }

                        }

        }



        static func encode(to container: inout SingleValueEncodingContainer, value: Any) throws {

                        if let value = value as? Bool {

                                        try container.encode(value)

                        } else if let value = value as? Int64 {

                                        try container.encode(value)

                        } else if let value = value as? Double {

                                        try container.encode(value)

                        } else if let value = value as? String {

                                        try container.encode(value)

                        } else if value is JSONNull {

                                        try container.encodeNil()

                        } else {

                                        throw encodingError(forValue: value, codingPath: container.codingPath)

                        }

        }



        public required init(from decoder: Decoder) throws {

                        if var arrayContainer = try? decoder.unkeyedContainer() {

                                        self.value = try JSONAny.decodeArray(from: &arrayContainer)

                        } else if var container = try? decoder.container(keyedBy: JSONCodingKey.self) {

                                        self.value = try JSONAny.decodeDictionary(from: &container)

                        } else {

                                        let container = try decoder.singleValueContainer()

                                        self.value = try JSONAny.decode(from: container)

                        }

        }



        public func encode(to encoder: Encoder) throws {

                        if let arr = self.value as? [Any] {

                                        var container = encoder.unkeyedContainer()

                                        try JSONAny.encode(to: &container, array: arr)

                        } else if let dict = self.value as? [String: Any] {

                                        var container = encoder.container(keyedBy: JSONCodingKey.self)

                                        try JSONAny.encode(to: &container, dictionary: dict)

                        } else {

                                        var container = encoder.singleValueContainer()

                                        try JSONAny.encode(to: &container, value: self.value)

                        }

        }

}







// MARK: - WebPage



/// Schema for a web page

/// Schema: `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0`

///

/// Example:

/// ```swift

/// let data = WebPage(

///     id: id

/// )

/// // Track as an event

/// Snowplow.defaultTracker()?.track(data.toEvent())

/// // Add as an entity to another event

/// let event = ScreenView(name: "Product")

/// event.entities.append(data.toEntity())

/// Snowplow.defaultTracker()?.track(event)

/// ```

struct WebPage {



        /// The page view ID.

        var id: String



        private var schema: String {

                return "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

        }



        private var payload: [String : Any] {

                var payload: [String : Any] = [:]

                payload["id"] = id

                return payload

        }



        /// Creates an event instance to be tracked by the tracker.

        func toEvent() -> SelfDescribing {

                return SelfDescribing(schema: schema, payload: payload)

        }



        /// Creates an entity that can be added to events.

        func toEntity() -> SelfDescribingJson {

                return SelfDescribingJson(schema: schema, andData: payload)

        }



}



// MARK: - Product



/// Schema for a product entity

/// Schema: `iglu:com.example/product/jsonschema/1-0-0`

///

/// Example:

/// ```swift

/// let data = Product(

///     category: category,

///     currency: currency,

///     id: id,

///     name: name,

///     price: price

/// )

/// // Track as an event

/// Snowplow.defaultTracker()?.track(data.toEvent())

/// // Add as an entity to another event

/// let event = ScreenView(name: "Product")

/// event.entities.append(data.toEntity())

/// Snowplow.defaultTracker()?.track(event)

/// ```

struct Product {



        /// The product category.

        var category: String

        /// The currency the product is listed in.

        var currency: String

        /// The product ID (SKU).

        var id: String

        /// The product name.

        var name: String

        /// The price of the product.

        var price: Double



        private var schema: String {

                return "iglu:com.example/product/jsonschema/1-0-0"

        }



        private var payload: [String : Any] {

                var payload: [String : Any] = [:]

                payload["category"] = category

                payload["currency"] = currency

                payload["id"] = id

                payload["name"] = name

                payload["price"] = price

                return payload

        }



        /// Creates an event instance to be tracked by the tracker.

        func toEvent() -> SelfDescribing {

                return SelfDescribing(schema: schema, payload: payload)

        }



        /// Creates an entity that can be added to events.

        func toEntity() -> SelfDescribingJson {

                return SelfDescribingJson(schema: schema, andData: payload)

        }



}
````

Here's how you could use the generated code:

```swift
import SnowplowTracker



/* Track a self-describing event */

_ = tracker.track(WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent())



/* Track an event with an entity attached */

let product = Product(

    category: "Snowplow/Shoes",

    currency: "EUR",

    id: "Product id",

    name: "Snowplow product",

    price: 10

)

let event = WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent()

event.entities.append(product.toEntity())

_ = tracker.track(event)
```

## Android (Kotlin)

Generated code for the [Android tracker](/docs/sources/mobile-trackers/). Snowtype produces Kotlin data classes with `toEvent()` and `toEntity()` methods.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-android-tracker",

  "language": "kotlin",

  "outpath": "./app/src/main/java/snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

````kotlin
// Automatically generated by Snowtype



package Snowtype



import com.snowplowanalytics.snowplow.event.SelfDescribing

import com.snowplowanalytics.snowplow.payload.SelfDescribingJson



/**

 * Schema for a web page

 * Schema: `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0`

 *

 * Example:

 * ```kotlin

 * val data = WebPage(

 *     id = id

 * )

 * // Track as an event

 * Snowplow.defaultTracker?.track(data.toEvent())

 * // Add as an entity to another event

 * val event = ScreenView(name = "Product")

 * event.entities.add(data.toEntity())

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

data class WebPage(



        /** The page view ID. */

        var id: String

) {

        private val schema = "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"



        private val payload: Map<String, Any?>

                get() {

                        val payload = HashMap<String, Any?>()

                        payload["id"] = id

                        return payload

                }



        /** Creates an event instance to be tracked by the tracker. */

        fun toEvent() : SelfDescribing {

                return SelfDescribing(schema, payload)

        }



        /** Creates an entity that can be added to events. */

        fun toEntity() : SelfDescribingJson {

                return SelfDescribingJson(schema, payload)

        }

}



/**

 * Schema for a product entity

 * Schema: `iglu:com.example/product/jsonschema/1-0-0`

 *

 * Example:

 * ```kotlin

 * val data = Product(

 *     category = category,

 *     currency = currency,

 *     id = id,

 *     name = name,

 *     price = price

 * )

 * // Track as an event

 * Snowplow.defaultTracker?.track(data.toEvent())

 * // Add as an entity to another event

 * val event = ScreenView(name = "Product")

 * event.entities.add(data.toEntity())

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

data class Product(



        /** The product category. */

        var category: String,



        /** The currency the product is listed in. */

        var currency: String,



        /** The product ID (SKU). */

        var id: String,



        /** The product name. */

        var name: String,



        /** The price of the product. */

        var price: Double

) {

        private val schema = "iglu:com.example/product/jsonschema/1-0-0"



        private val payload: Map<String, Any?>

                get() {

                        val payload = HashMap<String, Any?>()

                        payload["category"] = category

                        payload["currency"] = currency

                        payload["id"] = id

                        payload["name"] = name

                        payload["price"] = price

                        return payload

                }



        /** Creates an event instance to be tracked by the tracker. */

        fun toEvent() : SelfDescribing {

                return SelfDescribing(schema, payload)

        }



        /** Creates an entity that can be added to events. */

        fun toEntity() : SelfDescribingJson {

                return SelfDescribingJson(schema, payload)

        }

}
````

Here's how you could use the generated code:

```kotlin
import {{ specified package }}.Product

import {{ specified package }}.WebPage



/* Track a self-describing event */

tracker.track(WebPage(id = "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent())



/* Track an event with an entity attached */

val product = Product(

    id = "Product id",

    name = "Snowplow product",

    currency = "EUR",

    price = 10.0,

    category = "Snowplow/Shoes",

)

val event = WebPage(id = "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent()

event.entities.add(product.toEntity())

tracker.track(event)
```

## React Native (TypeScript)

Generated code for the [React Native tracker](/docs/sources/react-native-tracker/). Snowtype produces TypeScript types with `track` and `create` functions.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/react-native-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```typescript
import { ReactNativeTracker } from '@snowplow/react-native-tracker';

// Automatically generated by Snowtype



/**

 * Schema for a web page

 */

export type WebPage = {

    /**

     * The page view ID.

     */

    id: string;

}



/**

 * Schema for a product entity

 */

export type Product = {

    /**

     * The product category.

     */

    category: string;

    /**

     * The currency the product is listed in.

     */

    currency: string;

    /**

     * The product ID (SKU).

     */

    id: string;

    /**

     * The product name.

     */

    name: string;

    /**

     * The price of the product.

     */

    price: number;

}



interface CommonEventProperties<T = Record<string, unknown>> {

    /** Add context to an event by setting an Array of Self Describing JSON */

    contexts?: Array<SelfDescribing<T>>;

}



type Contexts<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribing<T>[] }



/**

 * SelfDescribing type

 */

export type SelfDescribing<T extends Record<keyof T, unknown> = Record<string, unknown>> = {

        /**

         * The schema string

         * @example 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0'

         */

        schema: string;

        /**

         * The data object which should conform to the supplied schema

         */

        data: T;

};



/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 */

export function trackWebPage<T extends {} = any>(tracker: ReactNativeTracker, webPage: WebPage & Contexts<T>){

    const { contexts, ...data } = webPage;

    tracker.trackSelfDescribingEvent({

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data

    }, contexts);

}



/**

 * Creates a Snowplow WebPage entity.

 */

export function createWebPage(webPage: WebPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 */

export function trackProduct<T extends {} = any>(tracker: ReactNativeTracker, product: Product & Contexts<T>){

    const { contexts, ...data } = product;

    tracker.trackSelfDescribingEvent({

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data

    }, contexts);

}



/**

 * Creates a Snowplow Product entity.

 */

export function createProduct(product: Product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

Here's how you could use the generated code:

```tsx
import {

  trackWebPage,

  createProduct,

  WebPage,

  Product,

  createWebPage,

} from "./src/tracking/snowplow";



/*

 * `t` is the tracker instance created by the

 * `createTracker` function of @snowplow/react-native-tracker.

 */



/* Track a self-describing event */

trackWebPage(t, { id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const product = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage(t, {

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product],

});



/* Enforce specific entity types using type arguments */

const webPage = createWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

});

trackWebPage<Product | WebPage>(t, {

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product, webPage],

});
```

## Flutter (Dart)

Generated code for the [Flutter tracker](/docs/sources/flutter-tracker/). Snowtype produces immutable Dart classes that implement `SelfDescribing`, with `schema`, `data`, and `toMap()` accessors.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-flutter-tracker",

  "language": "dart",

  "outpath": "./lib/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```dart
import 'package:flutter/foundation.dart';

import 'package:snowplow_tracker/events/self_describing.dart';



/// Schema for a web page

/// Schema: `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0`

@immutable

class WebPage implements SelfDescribing {



    /// The page view ID.

    final String id;



    const WebPage({

        required this.id

    });



    @override

    String endpoint() {

        return 'trackSelfDescribing';

    }



    @override

    Map<String, Object?> get data {

        return {

            'id': id,

        };

    }



    @override

    String get schema => 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0';



    @override

    Map<String, Object?> toMap() {

        return {

            'schema': schema,

            'data': data,

        };

    }

}

/// Schema for a product entity

/// Schema: `iglu:com.example/product/jsonschema/1-0-0`

@immutable

class Product implements SelfDescribing {



    /// The product category.

    final String category;



    /// The currency the product is listed in.

    final String currency;



    /// The product ID (SKU).

    final String id;



    /// The product name.

    final String name;



    /// The price of the product.

    final double price;



    const Product({

        required this.category,

        required this.currency,

        required this.id,

        required this.name,

        required this.price

    });



    @override

    String endpoint() {

        return 'trackSelfDescribing';

    }



    @override

    Map<String, Object?> get data {

        return {

            'category': category,

            'currency': currency,

            'id': id,

            'name': name,

            'price': price,

        };

    }



    @override

    String get schema => 'iglu:com.example/product/jsonschema/1-0-0';



    @override

    Map<String, Object?> toMap() {

        return {

            'schema': schema,

            'data': data,

        };

    }

}
```

Here's how you could use the generated code:

```dart
import './lib/tracking/snowplow/snowtype.dart';



/* Track a self-describing event */

await tracker.track(const WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43"));



/* Track an event with an entity attached */

const product = Product(

  category: "Snowplow/Shoes",

  currency: "EUR",

  id: "Product id",

  price: 10.0,

  name: "Snowplow product"

);

const event = WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43");

await tracker.track(event, contexts: [product]);
```

## Node.js

Snowtype can generate code in TypeScript or JavaScript for the [Node.js tracker](/docs/sources/node-js-tracker/).

### TypeScript

Snowtype generates TypeScript type definitions and typed `track` and `create` functions that import directly from `@snowplow/node-tracker`.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/node-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```typescript
import { buildSelfDescribingEvent, SelfDescribingJson, Timestamp, Tracker } from '@snowplow/node-tracker';

// Automatically generated by Snowtype



/**

 * Schema for a web page

 */

export type WebPage = {

    /**

     * The page view ID.

     */

    id: string;

}



/**

 * Schema for a product entity

 */

export type Product = {

    /**

     * The product category.

     */

    category: string;

    /**

     * The currency the product is listed in.

     */

    currency: string;

    /**

     * The product ID (SKU).

     */

    id: string;

    /**

     * The product name.

     */

    name: string;

    /**

     * The price of the product.

     */

    price: number;

}



interface CommonEventProperties<T = Record<string, unknown>> {

    /** Add context to an event by setting an Array of Self Describing JSON */

    context?: Array<SelfDescribingJson<T>> | null;

    /** Set the true timestamp or overwrite the device sent timestamp on an event */

    timestamp?: Timestamp | null;

}



type ContextsOrTimestamp<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribingJson<T>[] | null | undefined }



/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 */

export function trackWebPage<T extends {} = any>(tracker: Tracker, webPage: WebPage & ContextsOrTimestamp<T>){

    const { context, timestamp, ...data } = webPage;

    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow WebPage entity.

 */

export function createWebPage(webPage: WebPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 */

export function trackProduct<T extends {} = any>(tracker: Tracker, product: Product & ContextsOrTimestamp<T>){

    const { context, timestamp, ...data } = product;

    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.example/product/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow Product entity.

 */

export function createProduct(product: Product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

Here's how you could use the generated code in a Node.js project:

```tsx
import {

  trackWebPage,

  createProduct,

  WebPage,

  Product,

  createWebPage,

} from "./src/tracking/snowplow";



/*

 * `t` is the tracker instance created by the

 * `tracker` function of @snowplow/node-tracker.

 */



/* Track a self-describing event */

trackWebPage(t, { id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const product = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage(t, {

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product],

});



/* Enforce specific entity types using type arguments */

const webPage = createWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

});

trackWebPage<Product | WebPage>(t, {

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product, webPage],

});
```

### JavaScript

Snowtype generates JSDoc typedefs and untyped `track` and `create` functions, with a `@typedef {object} Tracker` for the tracker instance parameter.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/node-tracker",

  "language": "javascript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```javascript
import { buildSelfDescribingEvent } from '@snowplow/node-tracker';

// Automatically generated by Snowtype



/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Schema for a web page

 * @typedef {object} WebPage

 * @property {string} id The page view ID.

 */





/**

 * Typedef for a WebPage entity context.

 * @typedef {object} WebPageContext

 * @property {string} schema The schema of the context.

 * @property { WebPage } data The data to send for the context.

 */



/**

 * Schema for a product entity

 * @typedef {object} Product

 * @property {string} category The product category.

 * @property {string} currency The currency the product is listed in.

 * @property {string} id The product ID (SKU).

 * @property {string} name The product name.

 * @property {number} price The price of the product.

 */





/**

 * Typedef for a Product entity context.

 * @typedef {object} ProductContext

 * @property {string} schema The schema of the context.

 * @property { Product } data The data to send for the context.

 */



/**

 * Typedef for a Node.js tracker object.

 * @typedef {object} Tracker

 */



/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 * @param {Tracker} tracker - Tracker to send the event to.

 * @param { WebPage & Timestamp & Context } webPage - Attributes for WebPage.

 */

export function trackWebPage(tracker, webPage){

    const { context, timestamp, ...data } = webPage;



    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow WebPage entity.

 * @param { WebPage } webPage - Attributes for WebPage.

 */

export function createWebPage(webPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 * @param {Tracker} tracker - Tracker to send the event to.

 * @param { Product & Timestamp & Context } product - Attributes for Product.

 */

export function trackProduct(tracker, product){

    const { context, timestamp, ...data } = product;



    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.example/product/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow Product entity.

 * @param { Product } product - Attributes for Product.

 */

export function createProduct(product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

Here's how you could use the generated code in a Node.js project:

```javascript
import {

  trackWebPage,

  createProduct,

  createWebPage,

} from "./src/tracking/snowplow";



/*

 * `t` is the tracker instance created by the

 * `tracker` function of @snowplow/node-tracker.

 */



/* Track a self-describing event */

trackWebPage(t, { id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const product = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage(t, {

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [product],

});
```

## Golang

Generated code for the [Golang tracker](/docs/sources/golang-tracker/). Snowtype produces Go structs with `mapstructure` tags and `SnowplowFormat()` methods that implement the `SnowplowApplicable` interface, plus `Track` functions that accept a `*sp.Tracker`.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-golang-tracker",

  "language": "go",

  "outpath": "./tracking/snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```go
package snowtypegenerated



import "github.com/mitchellh/mapstructure"

import sp "github.com/snowplow/snowplow-golang-tracker/v3/tracker"



// Schema for a web page

type WebPage struct {

	// The page view ID.

	ID string `mapstructure:"id"`

}



// Schema for a product entity

type Product struct {

	// The product category.

	Category string  `mapstructure:"category"`

	// The currency the product is listed in.

	Currency string  `mapstructure:"currency"`

	// The product ID (SKU).

	ID       string  `mapstructure:"id"`

	// The product name.

	Name     string  `mapstructure:"name"`

	// The price of the product.

	Price    float64 `mapstructure:"price"`

}



// SnowplowApplicable signifies ability to be formatted as self-describing json

type SnowplowApplicable interface {

	// SnowplowFormat constructs a Snowplow SelfDescribingJson

	SnowplowFormat() (*sp.SelfDescribingJson, error)

}



// internal type to provide functional options for the API

type option func(*sp.SelfDescribingEvent) error



// WithTimestamp returns an option to set the device timestamp

func WithTimestamp(tstamp int64) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.Timestamp = &tstamp

		return nil

	}

}



// WithEventID returns an option to set the event_id

func WithEventID(id string) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.EventId = &id

		return nil

	}

}



// WithTrueTimestamp returns an option to set the true timestamp

func WithTrueTimestamp(tstamp int64) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.TrueTimestamp = &tstamp

		return nil

	}

}



// WithContexts returns an option to set the contexts of an event

func WithContexts(ctx ...SnowplowApplicable) option {

	return func(sde *sp.SelfDescribingEvent) error {

		finalCtx := make([]sp.SelfDescribingJson, len(ctx))

		var ctxErr error

		for i, val := range ctx {

			ctxSdj, ctxErr := val.SnowplowFormat()

			if ctxErr != nil {

				break

			}

			finalCtx[i] = *ctxSdj

		}

		if ctxErr != nil {

			return ctxErr

		}

		sde.Contexts = finalCtx

		return nil

	}

}



// WithSubject returns an option to set the Subject

func WithSubject(subject *sp.Subject) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.Subject = subject

		return nil

	}

}



// SnowplowFormat implements SnowplowApplicable

func (webPage WebPage) SnowplowFormat() (*sp.SelfDescribingJson, error) {

	var decodedMap map[string]interface{}

	err := mapstructure.Decode(webPage, &decodedMap)

	if err != nil {

		return nil, err

	}



	sdj := sp.InitSelfDescribingJson("iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0", decodedMap)

	return sdj, nil

}



// TrackWebPage tracks a WebPage

func TrackWebPage(tracker *sp.Tracker, webPage WebPage, opts ...option) error {

	sdj, err := webPage.SnowplowFormat()

	if err != nil {

		return err

	}



	sde := &sp.SelfDescribingEvent{Event: sdj}

	for _, opt := range opts {

		opt(sde)

	}



	tracker.TrackSelfDescribingEvent(*sde)

	return nil

}



// SnowplowFormat implements SnowplowApplicable

func (product Product) SnowplowFormat() (*sp.SelfDescribingJson, error) {

	var decodedMap map[string]interface{}

	err := mapstructure.Decode(product, &decodedMap)

	if err != nil {

		return nil, err

	}



	sdj := sp.InitSelfDescribingJson("iglu:com.example/product/jsonschema/1-0-0", decodedMap)

	return sdj, nil

}



// TrackProduct tracks a Product

func TrackProduct(tracker *sp.Tracker, product Product, opts ...option) error {

	sdj, err := product.SnowplowFormat()

	if err != nil {

		return err

	}



	sde := &sp.SelfDescribingEvent{Event: sdj}

	for _, opt := range opts {

		opt(sde)

	}



	tracker.TrackSelfDescribingEvent(*sde)

	return nil

}
```

Here's how you could use the generated code in a Go project:

```go
// Track a self-describing event

TrackWebPage(tracker, WebPage{ID: "212a9b63-1af7-4e96-9f35-e2fca110ff43"})



// Track an event with an entity attached

productName := "Snowplow product"

product := Product{

	ID:       "Product_id",

	Currency: "EUR",

	Price:    10,

	Category: "Snowplow/Shoes",

	Name:     &productName,

}



TrackWebPage(

	tracker,

	WebPage{ID: "212a9b63-1af7-4e96-9f35-e2fca110ff43"},

	WithContexts(product),

)
```

## Java

Generated code for the [Java tracker](/docs/sources/java-tracker/). Snowtype produces Java classes with private fields, getter and setter methods, and Javadoc comments. Each class is generated as a separate file.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-java-tracker",

  "language": "java",

  "outpath": "./src/main/java/snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```java
// WebPage.java



// Automatically generated by Snowtype



/**

 * Schema for a web page

 */

public class WebPage {

    private String id;



    /**

     * The page view ID.

     */

    public String getID() { return id; }

    public void setID(String value) { this.id = value; }

}
```

```java
// Product.java



// Automatically generated by Snowtype



/**

 * Schema for a product entity

 */

public class Product {

    private String category;

    private String currency;

    private String id;

    private String name;

    private double price;



    /**

     * The product category.

     */

    public String getCategory() { return category; }

    public void setCategory(String value) { this.category = value; }



    /**

     * The currency the product is listed in.

     */

    public String getCurrency() { return currency; }

    public void setCurrency(String value) { this.currency = value; }



    /**

     * The product ID (SKU).

     */

    public String getID() { return id; }

    public void setID(String value) { this.id = value; }



    /**

     * The product name.

     */

    public String getName() { return name; }

    public void setName(String value) { this.name = value; }



    /**

     * The price of the product.

     */

    public double getPrice() { return price; }

    public void setPrice(double value) { this.price = value; }

}
```

Here's how you could use the generated code in a Java project:

```java
import com.snowplowanalytics.snowplow.tracker.*;

import com.snowplowanalytics.snowplow.snowtype.*;

import com.snowplowanalytics.snowplow.tracker.events.SelfDescribing;



import java.util.Collections;



/* Track a self-describing event */

tracker.track(

    SelfDescribing.builder()

        .eventData(new WebPage.Builder()

            .setId("212a9b63-1af7-4e96-9f35-e2fca110ff43")

            .build()

            .toSelfDescribingJson())

        .build()

);



/* Track an event with an entity attached */

Product product = new Product.Builder()

    .setId("Product id")

    .setName("Snowplow product")

    .setCurrency("EUR")

    .setPrice(10.0)

    .setCategory("Snowplow/Shoes")

    .build();

WebPage webPage = new WebPage.Builder()

    .setId("212a9b63-1af7-4e96-9f35-e2fca110ff43")

    .build();

SelfDescribing event = SelfDescribing.builder()

    .eventData(product.toSelfDescribingJson())

    .customContext(Collections.singletonList(

        webPage.toSelfDescribingJson()))

    .build();

tracker.track(event);
```

## Google Tag Manager (JavaScript)

Generated code for [Google Tag Manager](/docs/sources/google-tag-manager/). Snowtype produces a single function that assigns tracking methods to `window.__snowtype`, designed to be used in a GTM [Custom JavaScript Variable](/docs/event-studio/implement-tracking/working-with-gtm/).

Snowtype will also generate a minified version.

```json
{

  "orgId": "your-org-id",

  "tracker": "google-tag-manager",

  "language": "javascript-gtm",

  "outpath": "./gtm",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```javascript
// Automatically generated by Snowtype

// This code should be copied and pasted in a Google Tag Manager Custom JavaScript Variable



function snowtypeInit() {

    function _objectWithoutProperties(e, t) { if (null == e) return {}; var o, r, i = _objectWithoutPropertiesLoose(e, t); if (Object.getOwnPropertySymbols) { var n = Object.getOwnPropertySymbols(e); for (r = 0; r < n.length; r++) o = n[r], t.indexOf(o) >= 0 || {}.propertyIsEnumerable.call(e, o) && (i[o] = e[o]); } return i; }

    function _objectWithoutPropertiesLoose(r, e) { if (null == r) return {}; var t = {}; for (var n in r) if ({}.hasOwnProperty.call(r, n)) { if (e.indexOf(n) >= 0) continue; t[n] = r[n]; } return t; }



    window.__snowtype = {

        /**

        * Track a Snowplow event for WebPage.

        * Schema for a web page

        * @param { WebPage & Timestamp & Context } webPage - Attributes for WebPage.

        * @param {string[]} [trackers] - Tracker names to send the event to.

        */

        trackWebPage: function(webPage, trackers){

                var trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

                var context = webPage.context;

                var timestamp = webPage.timestamp;

                var data = _objectWithoutProperties(webPage, ["context", "timestamp"]);

                window.snowplow('trackSelfDescribingEvent' + trackerNames, {

                        event: {

                                schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

                                data: data

                        },

                        context: context,

                        timestamp: timestamp,

                });

        },

        /**

        * Creates a Snowplow WebPage entity.

        * @param { WebPage } webPage - Attributes for WebPage.

        */

        createWebPage: function(webPage){

                return {

                        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

                        data: webPage

                };

        },

        /**

        * Track a Snowplow event for Product.

        * Schema for a product entity

        * @param { Product & Timestamp & Context } product - Attributes for Product.

        * @param {string[]} [trackers] - Tracker names to send the event to.

        */

        trackProduct: function(product, trackers){

                var trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

                var context = product.context;

                var timestamp = product.timestamp;

                var data = _objectWithoutProperties(product, ["context", "timestamp"]);

                window.snowplow('trackSelfDescribingEvent' + trackerNames, {

                        event: {

                                schema: 'iglu:com.example/product/jsonschema/1-0-0',

                                data: data

                        },

                        context: context,

                        timestamp: timestamp,

                });

        },

        /**

        * Creates a Snowplow Product entity.

        * @param { Product } product - Attributes for Product.

        */

        createProduct: function(product){

                return {

                        schema: 'iglu:com.example/product/jsonschema/1-0-0',

                        data: product

                };

        },

    };



    return window.__snowtype;

}
```

---

# Example Snowtype generated code for an event specification
> Example configuration files and generated output for each Snowtype-supported tracker and language.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/generate-tracking-code/example-event-specs/

The following examples show the Snowtype [configuration file](/docs/event-studio/implement-tracking/snowtype-config/), generated output, and how to use them, for a `User Log In` event specification. The event specification has a `login` event data structure, plus two entities: `user` and `user_authentication`.

The `tracker` and `language` fields in your configuration determine the language and structure of the output.

These are the example schemas:

### `login`

**Type:** Schema

Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

**Properties:**

| Property               | Description                                                                                                     |
| ---------------------- | --------------------------------------------------------------------------------------------------------------- |
| `method` _string_      | _Required._ The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.                     |
| `error_code` _string_  | _Optional._ An optional error code if the login attempt failed (e.g., 'invalid\_password', 'user\_not\_found'). |
| `is_success` _boolean_ | _Required._ Whether the login attempt was successful.                                                           |

### `user`

**Type:** Schema

A subset of the user fields from the users-service

**Properties:**

| Property                  | Description                                                                   |
| ------------------------- | ----------------------------------------------------------------------------- |
| `userId` _string_         | _Required._ The users UUID.                                                   |
| `firstName` _string_      | _Required._ The users first name.                                             |
| `lastName` _string_       | _Required._ The users last name.                                              |
| `organizationId` _string_ | _Required._ The organization's UUID that the user belongs to.                 |
| `email` _string_          | _Optional._ The users email address.                                          |
| `jobTitle` _string_       | _Optional._ The users job title.                                              |
| `accessLevel` _string_    | _Optional._ The users access level. Must be one of: `Admin`, `User`, `Custom` |

### `user_authentication`

**Type:** Schema

Data structure that defines how the user logged in

**Properties:**

| Property             | Description                                                                            |
| -------------------- | -------------------------------------------------------------------------------------- |
| `auth_type` _string_ | _Required._ Type of authentication used Must be one of: `google`, `facebook`, `manual` |

![Console screenshot showing the example event specification](/assets/images/example_event_spec_full-4de68d5b0428c3d71783b23c4c74d006.png)

## Web trackers

Snowtype can generate code in TypeScript or JavaScript, depending which version of the [web trackers](/docs/sources/web-trackers/) you're using.

### Browser (TypeScript)

Generated code for the [Browser tracker](/docs/sources/web-trackers/). Snowtype produces TypeScript type definitions with `track` and `create` functions that wrap `trackSelfDescribingEvent`, plus a `trackXSpec` function, in this case `trackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```typescript
import { trackSelfDescribingEvent, CommonEventProperties, SelfDescribingJson } from '@snowplow/browser-tracker';

// Automatically generated by Snowtype



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome

 * of the authentication attempt.

 */

export type Login = {

    /**

     * An optional error code if the login attempt failed (e.g., 'invalid_password',

     * 'user_not_found').

     */

    error_code?: null | string;

    /**

     * Whether the login attempt was successful.

     */

    is_success: boolean;

    /**

     * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

     */

    method: string;

}



/**

 * A subset of the user fields from the users-service

 */

export type User = {

    /**

     * The users access level.

     */

    accessLevel?: AccessLevel;

    /**

     * The users email address.

     */

    email?: null | string;

    /**

     * The users first name.

     */

    firstName: string;

    /**

     * The users job title.

     */

    jobTitle?: null | string;

    /**

     * The users last name.

     */

    lastName: string;

    /**

     * The organization's UUID that the user belongs to.

     */

    organizationId: string;

    /**

     * The users UUID.

     */

    userId: string;

}



/**

 * The users access level.

 */

export type AccessLevel = "Admin" | "User" | "Custom";



/**

 * Data structure that defines how the user logged in

 */

export type UserAuthentication = {

    /**

     * Type of authentication used

     */

    auth_type: AuthType;

}



/**

 * Type of authentication used

 */

export type AuthType = "google" | "facebook" | "manual";



/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification: Omit<EventSpecification, 'data_product_domain'> & Partial<Pick<EventSpecification, 'data_product_domain'>>){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}



/**

 * Automatically attached context for event specifications

 */

interface EventSpecification {

    id: string;

    version: number;

    name: string;

    data_product_id: string;

    data_product_name: string;

    data_product_domain?: string;

}



type ContextsOrTimestamp<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribingJson<T>[] | null | undefined }

/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 */

export function trackLogin<T extends {} = any>(login: Login & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = login;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow Login entity.

 */

export function createLogin(login: Login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 */

export function trackUser<T extends {} = any>(user: User & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = user;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow User entity.

 */

export function createUser(user: User){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 */

export function trackUserAuthentication<T extends {} = any>(userAuthentication: UserAuthentication & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = userAuthentication;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow UserAuthentication entity.

 */

export function createUserAuthentication(userAuthentication: UserAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 */

export function trackUserLogInSpec(userLogIn: Login & ContextsOrTimestamp<User | UserAuthentication>, trackers?: string[]){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



    const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        context,

    };



    trackLogin<User | UserAuthentication | EventSpecification>(modifiedUserLogIn, trackers);

}
```

Here's how you could use the generated code:

```typescript
import {

  trackUserLogInSpec,

  createUser,

  createUserAuthentication,

} from "./src/tracking/snowplow";



const user = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuth = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec({

  method: "google",

  is_success: true,

  context: [user, userAuth],

});
```

### Browser (JavaScript)

Generated code for the [Browser tracker](/docs/sources/web-trackers/) in JavaScript. Snowtype produces JSDoc typedefs, with `track` and `create` functions, plus a `trackXSpec` function, in this case `trackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "javascript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```javascript
import { trackSelfDescribingEvent } from '@snowplow/browser-tracker';

// Automatically generated by Snowtype



/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @typedef {object} Login

 * @property {string} method The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

 * @property {?string} [error_code] An optional error code if the login attempt failed (e.g., 'invalid_password', 'user_not_found').

 * @property {boolean} is_success Whether the login attempt was successful.

 */





/**

 * Typedef for a Login entity context.

 * @typedef {object} LoginContext

 * @property {string} schema The schema of the context.

 * @property { Login } data The data to send for the context.

 */



/**

 * A subset of the user fields from the users-service

 * @typedef {object} User

 * @property {string} userId The users UUID.

 * @property {string} firstName The users first name.

 * @property {string} lastName The users last name.

 * @property {string} organizationId The organization's UUID that the user belongs to.

 * @property {?string} [email] The users email address.

 * @property {?string} [jobTitle] The users job title.

 * @property {"Admin"|"User"|"Custom"|null} [accessLevel] The users access level.

 */





/**

 * Typedef for a User entity context.

 * @typedef {object} UserContext

 * @property {string} schema The schema of the context.

 * @property { User } data The data to send for the context.

 */



/**

 * Data structure that defines how the user logged in

 * @typedef {object} UserAuthentication

 * @property {"google"|"facebook"|"manual"} auth_type Type of authentication used

 */





/**

 * Typedef for a UserAuthentication entity context.

 * @typedef {object} UserAuthenticationContext

 * @property {string} schema The schema of the context.

 * @property { UserAuthentication } data The data to send for the context.

 */



/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}





/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @param { Login & Timestamp & Context } login - Attributes for Login.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

export function trackLogin(login, trackers){

    const { context, timestamp, ...data } = login;

    const event = {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow Login entity.

 * @param { Login } login - Attributes for Login.

 */

export function createLogin(login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 * @param { User & Timestamp & Context } user - Attributes for User.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

export function trackUser(user, trackers){

    const { context, timestamp, ...data } = user;

    const event = {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow User entity.

 * @param { User } user - Attributes for User.

 */

export function createUser(user){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 * @param { UserAuthentication & Timestamp & Context } userAuthentication - Attributes for UserAuthentication.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

export function trackUserAuthentication(userAuthentication, trackers){

    const { context, timestamp, ...data } = userAuthentication;

    const event = {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow UserAuthentication entity.

 * @param { UserAuthentication } userAuthentication - Attributes for UserAuthentication.

 */

export function createUserAuthentication(userAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Typedef for UserLogIn event specification contexts.

 * @typedef {object} UserLogInContexts

 * @property {(UserContext | UserAuthenticationContext)[]} [context] Contexts to include in the UserLogIn event specification.

 */



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 * @param { Login & Timestamp & UserLogInContexts } userLogIn - Attributes for UserLogIn event specification.

 * @param {string[]} [trackers] - Tracker names to send the event specification to.

 */

export function trackUserLogInSpec(userLogIn, trackers){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



    const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        context,

    };



    trackLogin(modifiedUserLogIn, trackers);

}
```

Here's how you could use the generated code:

```javascript
import {

  trackUserLogInSpec,

  createUser,

  createUserAuthentication,

} from "./src/tracking/snowplow";



const user = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuth = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec({

  method: "google",

  is_success: true,

  context: [user, userAuth],

});
```

### JavaScript tag

Generated code for the [JavaScript tag](/docs/sources/web-trackers/). Snowtype produces functions that call `window.snowplow()` directly, including a `trackXSpec` function, in this case `trackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/javascript-tracker",

  "language": "javascript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```javascript
// Automatically generated by Snowtype



/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @typedef {object} Login

 * @property {string} method The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

 * @property {?string} [error_code] An optional error code if the login attempt failed (e.g., 'invalid_password', 'user_not_found').

 * @property {boolean} is_success Whether the login attempt was successful.

 */





/**

 * Typedef for a Login entity context.

 * @typedef {object} LoginContext

 * @property {string} schema The schema of the context.

 * @property { Login } data The data to send for the context.

 */



/**

 * A subset of the user fields from the users-service

 * @typedef {object} User

 * @property {string} userId The users UUID.

 * @property {string} firstName The users first name.

 * @property {string} lastName The users last name.

 * @property {string} organizationId The organization's UUID that the user belongs to.

 * @property {?string} [email] The users email address.

 * @property {?string} [jobTitle] The users job title.

 * @property {"Admin"|"User"|"Custom"|null} [accessLevel] The users access level.

 */





/**

 * Typedef for a User entity context.

 * @typedef {object} UserContext

 * @property {string} schema The schema of the context.

 * @property { User } data The data to send for the context.

 */



/**

 * Data structure that defines how the user logged in

 * @typedef {object} UserAuthentication

 * @property {"google"|"facebook"|"manual"} auth_type Type of authentication used

 */





/**

 * Typedef for a UserAuthentication entity context.

 * @typedef {object} UserAuthenticationContext

 * @property {string} schema The schema of the context.

 * @property { UserAuthentication } data The data to send for the context.

 */



/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @param { Login & Timestamp & Context } login - Attributes for Login.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

function trackLogin(login, trackers){

    const trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

    const { context, timestamp, ...data } = login;

    window.snowplow('trackSelfDescribingEvent' + trackerNames, {

        event: {

            schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

            data

        },

        context,

        timestamp,

    });

}



/**

 * Creates a Snowplow Login entity.

 * @param { Login } login - Attributes for Login.

 */

function createLogin(login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 * @param { User & Timestamp & Context } user - Attributes for User.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

function trackUser(user, trackers){

    const trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

    const { context, timestamp, ...data } = user;

    window.snowplow('trackSelfDescribingEvent' + trackerNames, {

        event: {

            schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

            data

        },

        context,

        timestamp,

    });

}



/**

 * Creates a Snowplow User entity.

 * @param { User } user - Attributes for User.

 */

function createUser(user){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 * @param { UserAuthentication & Timestamp & Context } userAuthentication - Attributes for UserAuthentication.

 * @param {string[]} [trackers] - Tracker names to send the event to.

 */

function trackUserAuthentication(userAuthentication, trackers){

    const trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

    const { context, timestamp, ...data } = userAuthentication;

    window.snowplow('trackSelfDescribingEvent' + trackerNames, {

        event: {

            schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

            data

        },

        context,

        timestamp,

    });

}



/**

 * Creates a Snowplow UserAuthentication entity.

 * @param { UserAuthentication } userAuthentication - Attributes for UserAuthentication.

 */

function createUserAuthentication(userAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}



/**

 * Typedef for UserLogIn event specification contexts.

 * @typedef {object} UserLogInContexts

 * @property {(UserContext | UserAuthenticationContext)[]} [context] Contexts to include in the UserLogIn event specification.

 */



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 * @param { Login & Timestamp & UserLogInContexts } userLogIn - Attributes for UserLogIn event specification.

 * @param {string[]} [trackers] - Tracker names to send the event specification to.

 */

function trackUserLogInSpec(userLogIn, trackers){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



    const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        context,

    };



    trackLogin(modifiedUserLogIn, trackers);

}
```

Here's how you could use the generated code:

```javascript
const user = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuth = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec({

  method: "google",

  is_success: true,

  context: [user, userAuth],

});
```

## iOS (Swift)

Generated code for the [iOS tracker](/docs/sources/mobile-trackers/). Snowtype produces Swift structs with `toEvent()` and `toEntity()` methods, plus a `toXSpec()` extension on the event struct that bundles it with its required entities and attaches the event specification entity. In this case, a `toUserLogInSpec()` extension on the `Login` struct.

The generated code is compatible with both Swift 5.5+ and 6.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-ios-tracker",

  "language": "swift",

  "outpath": "./Sources/Tracking/Snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": [],

  // "namespace": "SnowplowTracking"

}
```

When generating tracking code for the iOS tracker, you can set the optional `namespace` option in your Snowtype configuration. All classes generated will be included in this namespace. This can be helpful for avoiding naming conflicts with other types in your codebase.

Snowtype generates `JSONNull`, `JSONCodingKey`, and `JSONAny` helper classes for encoding and decoding JSON values. These are omitted here for brevity.

````swift
import Foundation

import SnowplowTracker



// MARK: - Encode/decode helpers



// Helper classes JSONNull, JSONCodingKey, and JSONAny not shown



// MARK: - Login



/// Captured when a user attempts to log in to their account. Tracks the method and outcome

/// of the authentication attempt.

/// Schema: `iglu:com.snplow.msc.aws/login/jsonschema/1-0-0`

///

/// Example:

/// ```swift

/// let data = Login(

///     isSuccess: isSuccess,

///     method: method,

///     errorCode: errorCode

/// )

/// // Track as an event

/// Snowplow.defaultTracker()?.track(data.toEvent())

/// // Add as an entity to another event

/// let event = ScreenView(name: "Product")

/// event.entities.append(data.toEntity())

/// Snowplow.defaultTracker()?.track(event)

/// ```

struct Login {



        /// Whether the login attempt was successful.

        var isSuccess: Bool

        /// The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

        var method: String

        /// An optional error code if the login attempt failed (e.g., 'invalid_password',

        /// 'user_not_found').

        var errorCode: String?



        private var schema: String {

                return "iglu:com.snplow.msc.aws/login/jsonschema/1-0-0"

        }



        private var payload: [String : Any] {

                var payload: [String : Any] = [:]

                if let errorCode = errorCode {

                        payload["error_code"] = errorCode

                }

                payload["is_success"] = isSuccess

                payload["method"] = method

                return payload

        }



        /// Creates an event instance to be tracked by the tracker.

        func toEvent() -> SelfDescribing {

                return SelfDescribing(schema: schema, payload: payload)

        }



        /// Creates an entity that can be added to events.

        func toEntity() -> SelfDescribingJson {

                return SelfDescribingJson(schema: schema, andData: payload)

        }



}



// MARK: - User



/// A subset of the user fields from the users-service

/// Schema: `iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1`

///

/// Example:

/// ```swift

/// let data = User(

///     firstName: firstName,

///     lastName: lastName,

///     organizationID: organizationID,

///     userID: userID,

///     accessLevel: accessLevel,

///     email: email,

///     jobTitle: jobTitle

/// )

/// // Track as an event

/// Snowplow.defaultTracker()?.track(data.toEvent())

/// // Add as an entity to another event

/// let event = ScreenView(name: "Product")

/// event.entities.append(data.toEntity())

/// Snowplow.defaultTracker()?.track(event)

/// ```

struct User {



        /// The users first name.

        var firstName: String

        /// The users last name.

        var lastName: String

        /// The organization's UUID that the user belongs to.

        var organizationID: String

        /// The users UUID.

        var userID: String

        /// The users access level.

        var accessLevel: AccessLevel?

        /// The users email address.

        var email: String?

        /// The users job title.

        var jobTitle: String?



        private var schema: String {

                return "iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1"

        }



        private var payload: [String : Any] {

                var payload: [String : Any] = [:]

                if let accessLevel = accessLevel {

                        payload["accessLevel"] = accessLevel.rawValue

                }

                if let email = email {

                        payload["email"] = email

                }

                payload["firstName"] = firstName

                if let jobTitle = jobTitle {

                        payload["jobTitle"] = jobTitle

                }

                payload["lastName"] = lastName

                payload["organizationId"] = organizationID

                payload["userId"] = userID

                return payload

        }



        /// Creates an event instance to be tracked by the tracker.

        func toEvent() -> SelfDescribing {

                return SelfDescribing(schema: schema, payload: payload)

        }



        /// Creates an entity that can be added to events.

        func toEntity() -> SelfDescribingJson {

                return SelfDescribingJson(schema: schema, andData: payload)

        }



}



// MARK: - UserAuthentication



/// Data structure that defines how the user logged in

/// Schema: `iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0`

///

/// Example:

/// ```swift

/// let data = UserAuthentication(

///     authType: authType

/// )

/// // Track as an event

/// Snowplow.defaultTracker()?.track(data.toEvent())

/// // Add as an entity to another event

/// let event = ScreenView(name: "Product")

/// event.entities.append(data.toEntity())

/// Snowplow.defaultTracker()?.track(event)

/// ```

struct UserAuthentication {



        /// Type of authentication used

        var authType: AuthType



        private var schema: String {

                return "iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0"

        }



        private var payload: [String : Any] {

                var payload: [String : Any] = [:]

                payload["auth_type"] = authType.rawValue

                return payload

        }



        /// Creates an event instance to be tracked by the tracker.

        func toEvent() -> SelfDescribing {

                return SelfDescribing(schema: schema, payload: payload)

        }



        /// Creates an entity that can be added to events.

        func toEntity() -> SelfDescribingJson {

                return SelfDescribingJson(schema: schema, andData: payload)

        }



}



// MARK: - EventSpecification



/// Entity schema for referencing an event specification

/// Schema: `iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4`

///

/// Example:

/// ```swift

/// let data = EventSpecification(

///     id: id,

///     version: version,

///     name: name,

///     dataProductId: dataProductId,

///     dataProductName: dataProductName,

///     dataProductDomain: dataProductDomain

/// )

/// // Track as an event

/// Snowplow.defaultTracker()?.track(data.toEvent())

/// // Add as an entity to another event

/// let event = ScreenView(name: "Product")

/// event.entities.append(data.toEntity())

/// Snowplow.defaultTracker()?.track(event)

/// ```

struct EventSpecification {



        /// Identifier for the event specification that the event adheres

        var id: String

        /// Version of the event specification that the event adheres to

        var version: Int

        /// Name for the event specification that the event adheres to

        var name: String

        /// Identifier for the data product that the event specification belongs to

        var dataProductId: String

        /// Name for the data product that the event specification belongs to

        var dataProductName: String

        /// Domain for the data product that the event specification belongs to

        var dataProductDomain: String?



        private var schema: String {

                return "iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4"

        }



        private var payload: [String : Any] {

                var payload: [String : Any] = [:]

                payload["id"] = id

                payload["version"] = version

                payload["name"] = name

                payload["data_product_id"] = dataProductId

                payload["data_product_name"] = dataProductName

                if let dataProductDomain = dataProductDomain {

                        payload["data_product_domain"] = dataProductDomain

                }

                return payload

        }



        /// Creates an event instance to be tracked by the tracker.

        func toEvent() -> SelfDescribing {

                return SelfDescribing(schema: schema, payload: payload)

        }



        /// Creates an entity that can be added to events.

        func toEntity() -> SelfDescribingJson {

                return SelfDescribingJson(schema: schema, andData: payload)

        }



}



// MARK: - AccessLevel



/// The users access level.

enum AccessLevel: String {

        case admin = "Admin"

        case custom = "Custom"

        case user = "User"

}



// MARK: - AuthType



/// Type of authentication used

enum AuthType: String {

        case facebook = "facebook"

        case google = "google"

        case manual = "manual"

}



extension Login {

        /// Creates an event with entities for a UserLogIn event specification.

        /// ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

        /// Example:

        /// ```swift

        /// let data = Login(...)

        /// let dataUser = User(...)

        /// let dataUserAuthentication = UserAuthentication(...)

        /// let event = data.toUserLogInSpec(dataUser, dataUserAuthentication)

        /// // Track as an event

        /// Snowplow.defaultTracker()?.track(event)

        /// ```

        func toUserLogInSpec(_ entityUser: User, _ entityUserAuthentication: UserAuthentication) -> SelfDescribing {

                let event = toEvent()

                let eventSpec = EventSpecification(

                        id: "a965caf1-88a6-4a89-9aea-cc92516a9d56",

                        version: 8,

                        name: "User Log In",

                        dataProductId: "57471841-aa79-445d-b4f7-1cbd073a3188",

                        dataProductName: "Checkout Flow",

                        dataProductDomain: "Marketing"

                )

                event.entities.append(entityUser.toEntity())

                event.entities.append(entityUserAuthentication.toEntity())

                event.entities.append(eventSpec.toEntity())

                return event

        }

}
````

Here's how you could use the generated code:

```swift
import SnowplowTracker



let user = User(

    firstName: "Ada",

    lastName: "Lovelace",

    organizationID: "57471841-aa79-445d-b4f7-1cbd073a3188",

    userID: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

    email: "ada@example.com",

    accessLevel: .admin

)



let userAuth = UserAuthentication(authType: .google)



let login = Login(

    isSuccess: true,

    method: "google"

)



let event = login.toUserLogInSpec(user, userAuth)

Snowplow.defaultTracker()?.track(event)
```

## Android (Kotlin)

Generated code for the [Android tracker](/docs/sources/mobile-trackers/). Snowtype produces Kotlin data classes with `toEvent()` and `toEntity()` methods, plus a `toXSpec()` extension function for the event that bundles it with its required entities and attaches the event specification entity. In this case, a `toUserLogInSpec()` extension on `Login`.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-android-tracker",

  "language": "kotlin",

  "outpath": "./app/src/main/java/snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

````kotlin
// Automatically generated by Snowtype



package Snowtype



import com.snowplowanalytics.snowplow.event.SelfDescribing

import com.snowplowanalytics.snowplow.payload.SelfDescribingJson

import kotlinx.serialization.json.*



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome

 * of the authentication attempt.

 * Schema: `iglu:com.snplow.msc.aws/login/jsonschema/1-0-0`

 *

 * Example:

 * ```kotlin

 * val data = Login(

 *     isSuccess = isSuccess,

 *     method = method,

 *     errorCode = errorCode

 * )

 * // Track as an event

 * Snowplow.defaultTracker?.track(data.toEvent())

 * // Add as an entity to another event

 * val event = ScreenView(name = "Product")

 * event.entities.add(data.toEntity())

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

data class Login(



        /** Whether the login attempt was successful. */

        var isSuccess: Boolean,



        /** The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'. */

        var method: String,



        /**

         * An optional error code if the login attempt failed (e.g., 'invalid_password',

         * 'user_not_found').

         */

        var errorCode: String? = null

) {

        private val schema = "iglu:com.snplow.msc.aws/login/jsonschema/1-0-0"



        private val payload: Map<String, Any?>

                get() {

                        val payload = HashMap<String, Any?>()

                        errorCode?.let { payload["error_code"] = it }

                        payload["is_success"] = isSuccess

                        payload["method"] = method

                        return payload

                }



        /** Creates an event instance to be tracked by the tracker. */

        fun toEvent() : SelfDescribing {

                return SelfDescribing(schema, payload)

        }



        /** Creates an entity that can be added to events. */

        fun toEntity() : SelfDescribingJson {

                return SelfDescribingJson(schema, payload)

        }

}



/**

 * A subset of the user fields from the users-service

 * Schema: `iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1`

 *

 * Example:

 * ```kotlin

 * val data = User(

 *     firstName = firstName,

 *     lastName = lastName,

 *     organizationID = organizationID,

 *     userID = userID,

 *     accessLevel = accessLevel,

 *     email = email,

 *     jobTitle = jobTitle

 * )

 * // Track as an event

 * Snowplow.defaultTracker?.track(data.toEvent())

 * // Add as an entity to another event

 * val event = ScreenView(name = "Product")

 * event.entities.add(data.toEntity())

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

data class User(



        /** The users first name. */

        var firstName: String,



        /** The users last name. */

        var lastName: String,



        /** The organization's UUID that the user belongs to. */

        var organizationID: String,



        /** The users UUID. */

        var userID: String,



        /** The users access level. */

        var accessLevel: AccessLevel? = null,



        /** The users email address. */

        var email: String? = null,



        /** The users job title. */

        var jobTitle: String? = null

) {

        private val schema = "iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1"



        private val payload: Map<String, Any?>

                get() {

                        val payload = HashMap<String, Any?>()

                        accessLevel?.let { payload["accessLevel"] = it.string }

                        email?.let { payload["email"] = it }

                        payload["firstName"] = firstName

                        jobTitle?.let { payload["jobTitle"] = it }

                        payload["lastName"] = lastName

                        payload["organizationId"] = organizationID

                        payload["userId"] = userID

                        return payload

                }



        /** Creates an event instance to be tracked by the tracker. */

        fun toEvent() : SelfDescribing {

                return SelfDescribing(schema, payload)

        }



        /** Creates an entity that can be added to events. */

        fun toEntity() : SelfDescribingJson {

                return SelfDescribingJson(schema, payload)

        }

}



/**

 * Data structure that defines how the user logged in

 * Schema: `iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0`

 *

 * Example:

 * ```kotlin

 * val data = UserAuthentication(

 *     authType = authType

 * )

 * // Track as an event

 * Snowplow.defaultTracker?.track(data.toEvent())

 * // Add as an entity to another event

 * val event = ScreenView(name = "Product")

 * event.entities.add(data.toEntity())

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

data class UserAuthentication(



        /** Type of authentication used */

        var authType: AuthType

) {

        private val schema = "iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0"



        private val payload: Map<String, Any?>

                get() {

                        val payload = HashMap<String, Any?>()

                        payload["auth_type"] = authType.string

                        return payload

                }



        /** Creates an event instance to be tracked by the tracker. */

        fun toEvent() : SelfDescribing {

                return SelfDescribing(schema, payload)

        }



        /** Creates an entity that can be added to events. */

        fun toEntity() : SelfDescribingJson {

                return SelfDescribingJson(schema, payload)

        }

}



/**

 * Entity schema for referencing an event specification

 * Schema: `iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4`

 *

 * Example:

 * ```kotlin

 * val data = EventSpecification(

 *     id = id,

 *     version = version,

 *     name = name,

 *     dataProductId = dataProductId,

 *     dataProductName = dataProductName,

 *     dataProductDomain = dataProductDomain

 * )

 * // Track as an event

 * Snowplow.defaultTracker?.track(data.toEvent())

 * // Add as an entity to another event

 * val event = ScreenView(name = "Product")

 * event.entities.add(data.toEntity())

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

data class EventSpecification(



        /** Identifier for the event specification that the event adheres */

        var id: String,



        /** Version of the event specification that the event adheres to*/

        var version: Int,



        /** Name for the event specification that the event adheres to */

        var name: String,



        /** Identifier for the data product that the event specification belongs to */

        var dataProductId: String,



        /** Name for the data product that the event specification belongs to */

        var dataProductName: String,



        /** Domain for the data product that the event specification belongs to */

        var dataProductDomain: String? = null

) {

        private val schema = "iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4"



        private val payload: Map<String, Any?>

                get() {

                        val payload = HashMap<String, Any?>()

                        payload["id"] = id

                        payload["version"] = version

                        payload["name"] = name

                        payload["data_product_id"] = dataProductId

                        payload["data_product_name"] = dataProductName

                        dataProductDomain?.let { payload["data_product_domain"] = it }

                        return payload

                }



        /** Creates an event instance to be tracked by the tracker. */

        fun toEvent() : SelfDescribing {

                return SelfDescribing(schema, payload)

        }



        /** Creates an entity that can be added to events. */

        fun toEntity() : SelfDescribingJson {

                return SelfDescribingJson(schema, payload)

        }

}



/**

 * The users access level.

*/

enum class AccessLevel(val string: String) {

        Admin("Admin"),

        Custom("Custom"),

        User("User")

}



/**

 * Type of authentication used

*/

enum class AuthType(val string: String) {

        Facebook("facebook"),

        Google("google"),

        Manual("manual")

}



/**

 * Creates an event with entities for a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 * Example:

 * ```kotlin

 * val data = Login(...)

 * val dataUser = User(...)

 * val dataUserAuthentication = UserAuthentication(...)

 * val event = data.toUserLogInSpec(dataUser, dataUserAuthentication)

 * // Track as an event

 * Snowplow.defaultTracker?.track(event)

 * ```

*/

fun Login.toUserLogInSpec(entityUser: User, entityUserAuthentication: UserAuthentication) : SelfDescribing {

        val event = toEvent()

        val eventSpec = EventSpecification(

                "a965caf1-88a6-4a89-9aea-cc92516a9d56",

                8,

                "User Log In",

                "57471841-aa79-445d-b4f7-1cbd073a3188",

                "Checkout Flow",

                "Marketing"

        )

        event.entities.add(entityUser.toEntity())

        event.entities.add(entityUserAuthentication.toEntity())

        event.entities.add(eventSpec.toEntity())

        return event

}
````

Here's how you could use the generated code:

```kotlin
import Snowtype.Login

import Snowtype.User

import Snowtype.UserAuthentication

import Snowtype.toUserLogInSpec



val user = User(

    userID = "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

    firstName = "Ada",

    lastName = "Lovelace",

    organizationID = "57471841-aa79-445d-b4f7-1cbd073a3188",

    email = "ada@example.com",

    accessLevel = AccessLevel.Admin

)



val userAuth = UserAuthentication(authType = AuthType.Google)



val login = Login(

    method = "google",

    isSuccess = true

)



val event = login.toUserLogInSpec(user, userAuth)

Snowplow.defaultTracker?.track(event)
```

## React Native (TypeScript)

Generated code for the [React Native tracker](/docs/sources/react-native-tracker/). Snowtype produces TypeScript types with `track` and `create` functions, plus a `trackXSpec` function, in this case `trackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/react-native-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```typescript
import { ReactNativeTracker } from '@snowplow/react-native-tracker';

// Automatically generated by Snowtype



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome

 * of the authentication attempt.

 */

export type Login = {

    /**

     * An optional error code if the login attempt failed (e.g., 'invalid_password',

     * 'user_not_found').

     */

    error_code?: null | string;

    /**

     * Whether the login attempt was successful.

     */

    is_success: boolean;

    /**

     * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

     */

    method: string;

}



/**

 * A subset of the user fields from the users-service

 */

export type User = {

    /**

     * The users access level.

     */

    accessLevel?: AccessLevel;

    /**

     * The users email address.

     */

    email?: null | string;

    /**

     * The users first name.

     */

    firstName: string;

    /**

     * The users job title.

     */

    jobTitle?: null | string;

    /**

     * The users last name.

     */

    lastName: string;

    /**

     * The organization's UUID that the user belongs to.

     */

    organizationId: string;

    /**

     * The users UUID.

     */

    userId: string;

}



/**

 * The users access level.

 */

export type AccessLevel = "Admin" | "User" | "Custom";



/**

 * Data structure that defines how the user logged in

 */

export type UserAuthentication = {

    /**

     * Type of authentication used

     */

    auth_type: AuthType;

}



/**

 * Type of authentication used

 */

export type AuthType = "google" | "facebook" | "manual";



interface CommonEventProperties<T = Record<string, unknown>> {

    /** Add context to an event by setting an Array of Self Describing JSON */

    contexts?: Array<SelfDescribing<T>>;

}



type Contexts<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribing<T>[] }



/**

 * SelfDescribing type

 */

export type SelfDescribing<T extends Record<keyof T, unknown> = Record<string, unknown>> = {

        /**

         * The schema string

         * @example 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0'

         */

        schema: string;

        /**

         * The data object which should conform to the supplied schema

         */

        data: T;

};





/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification: Omit<EventSpecification, 'data_product_domain'> & Partial<Pick<EventSpecification, 'data_product_domain'>>){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}



/**

 * Automatically attached context for event specifications

 */

interface EventSpecification {

    id: string;

    version: number;

    name: string;

    data_product_id: string;

    data_product_name: string;

    data_product_domain?: string;

}



/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 */

export function trackLogin<T extends {} = any>(tracker: ReactNativeTracker, login: Login & Contexts<T>){

    const { contexts, ...data } = login;

    tracker.trackSelfDescribingEvent({

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data

    }, contexts);

}



/**

 * Creates a Snowplow Login entity.

 */

export function createLogin(login: Login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 */

export function trackUser<T extends {} = any>(tracker: ReactNativeTracker, user: User & Contexts<T>){

    const { contexts, ...data } = user;

    tracker.trackSelfDescribingEvent({

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data

    }, contexts);

}



/**

 * Creates a Snowplow User entity.

 */

export function createUser(user: User){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 */

export function trackUserAuthentication<T extends {} = any>(tracker: ReactNativeTracker, userAuthentication: UserAuthentication & Contexts<T>){

    const { contexts, ...data } = userAuthentication;

    tracker.trackSelfDescribingEvent({

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data

    }, contexts);

}



/**

 * Creates a Snowplow UserAuthentication entity.

 */

export function createUserAuthentication(userAuthentication: UserAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Tracks a UserLogIn event specification.

 */

export function trackUserLogInSpec(tracker: ReactNativeTracker, userLogIn: Login & Contexts<User | UserAuthentication>){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



    const contexts = Array.isArray(userLogIn.contexts)

        ? [...userLogIn.contexts, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        contexts,

    } as Login & Contexts<User | UserAuthentication>;



    trackLogin<User | UserAuthentication>(tracker, modifiedUserLogIn);

}
```

Here's how you could use the generated code:

```tsx
import {

  trackUserLogInSpec,

  createUser,

  createUserAuthentication,

} from "./src/tracking/snowplow";



/*

 * `t` is the tracker instance created by the

 * `createTracker` function of @snowplow/react-native-tracker.

 */



const user = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuth = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec(t, {

  method: "google",

  is_success: true,

  contexts: [user, userAuth],

});
```

## Flutter (Dart)

Generated code for the [Flutter tracker](/docs/sources/flutter-tracker/). Snowtype produces immutable Dart classes that implement `SelfDescribing`, with `schema`, `data`, and `toMap()` accessors, plus a `toXSpec()` extension function for the event that bundles it with its required entities and attaches the event specification entity. In this case, a `toUserLogInSpec()` extension on `Login`.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-flutter-tracker",

  "language": "dart",

  "outpath": "./lib/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

````dart
import 'package:flutter/foundation.dart';

import 'package:snowplow_tracker/events/self_describing.dart';



/// Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

/// Schema: `iglu:com.snplow.msc.aws/login/jsonschema/1-0-0`

@immutable

class Login implements SelfDescribing {



    /// An optional error code if the login attempt failed (e.g., 'invalid_password',,'user_not_found').

    final String? errorCode;



    /// Whether the login attempt was successful.

    final bool isSuccess;



    /// The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

    final String method;



    const Login({

        this.errorCode,

        required this.isSuccess,

        required this.method

    });



    @override

    String endpoint() {

        return 'trackSelfDescribing';

    }



    @override

    Map<String, Object?> get data {

        return {

            if (errorCode != null) 'error_code': errorCode,

            'is_success': isSuccess,

            'method': method,

        };

    }



    @override

    String get schema => 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0';



    @override

    Map<String, Object?> toMap() {

        return {

            'schema': schema,

            'data': data,

        };

    }

}

/// A subset of the user fields from the users-service

/// Schema: `iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1`

@immutable

class User implements SelfDescribing {



    /// The users access level.

    final AccessLevel? accessLevel;



    /// The users email address.

    final String? email;



    /// The users first name.

    final String firstName;



    /// The users job title.

    final String? jobTitle;



    /// The users last name.

    final String lastName;



    /// The organization's UUID that the user belongs to.

    final String organizationId;



    /// The users UUID.

    final String userId;



    const User({

        this.accessLevel,

        this.email,

        required this.firstName,

        this.jobTitle,

        required this.lastName,

        required this.organizationId,

        required this.userId

    });



    @override

    String endpoint() {

        return 'trackSelfDescribing';

    }



    @override

    Map<String, Object?> get data {

        return {

            if (accessLevel != null) 'accessLevel': accessLevel?.value,

            if (email != null) 'email': email,

            'firstName': firstName,

            if (jobTitle != null) 'jobTitle': jobTitle,

            'lastName': lastName,

            'organizationId': organizationId,

            'userId': userId,

        };

    }



    @override

    String get schema => 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1';



    @override

    Map<String, Object?> toMap() {

        return {

            'schema': schema,

            'data': data,

        };

    }

}

/// Data structure that defines how the user logged in

/// Schema: `iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0`

@immutable

class UserAuthentication implements SelfDescribing {



    /// Type of authentication used

    final AuthType authType;



    const UserAuthentication({

        required this.authType

    });



    @override

    String endpoint() {

        return 'trackSelfDescribing';

    }



    @override

    Map<String, Object?> get data {

        return {

            'auth_type': authType.value,

        };

    }



    @override

    String get schema => 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0';



    @override

    Map<String, Object?> toMap() {

        return {

            'schema': schema,

            'data': data,

        };

    }

}



/// The users access level.

enum AccessLevel {

    admin("Admin"),

    custom("Custom"),

    user("User");



    const AccessLevel(this.value);

    final String value;

}



/// Type of authentication used

enum AuthType {

    facebook("facebook"),

    google("google"),

    manual("manual");



    const AuthType(this.value);

    final String value;

}



/// For internal use.

@immutable

class EventSpecification {

    final SelfDescribing event;

    final List<SelfDescribing> contexts;



    const EventSpecification({

        required this.event,

        required this.contexts

    });

}



/// For internal use.

@immutable

class EventSpecEntity implements SelfDescribing {

    /// Identifier for the data product that the event specification belongs to

    final String? dataProductId;

    /// Name for the data product that the event specification belongs to

    final String? dataProductName;

    /// Domain of the data product that the event specification belongs to

    final String? dataProductDomain;

    /// Identifier for the event specification that the event adheres to

    final String id;

    /// Version of the event specification that the event adheres to

    final int version;

    /// Name for the event specification that the event adheres to

    final String? name;

    const EventSpecEntity({

        this.dataProductId,

        this.dataProductName,

        this.dataProductDomain,

        required this.id,

        required this.version,

        this.name

    });



    @override

    String endpoint() {

            return 'trackSelfDescribing';

    }



    @override

    Map<String, Object?> get data {

        return {

            if (dataProductId != null) 'data_product_id': dataProductId,

            if (dataProductName != null) 'data_product_name': dataProductName,

            if (dataProductDomain != null) 'data_product_domain': dataProductDomain,

            'id': id,

            'version': version,

            if (name != null) 'name': name,

        };

    }



    @override

    String get schema => 'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4';



    @override

    Map<String, Object?> toMap() {

        return {

            'schema': schema,

            'data': data,

        };

    }

}



/// Creates an event with entities for a UserLogIn event specification.

/// ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

/// Example:

/// ```dart

/// const baseEvent = Login(...);

/// final eventSpec = baseEvent.toUserLogInSpec(

///   dataUser: User(...), dataUserAuthentication: UserAuthentication(...)

/// );

/// /// Track as an event and list of context entities

/// tracker.track(eventSpec.event, contexts: eventSpec.contexts);

/// ```

extension UserLogInEventSpec on Login {

    EventSpecification toUserLogInSpec({ required User dataUser, required UserAuthentication dataUserAuthentication }) {

        final entities = <SelfDescribing>[];

        const eventSpecEntity = EventSpecEntity(

            id: "a965caf1-88a6-4a89-9aea-cc92516a9d56",

            version: 8,

            name: "User Log In",

            dataProductId: "57471841-aa79-445d-b4f7-1cbd073a3188",

            dataProductName: "Checkout Flow",

            dataProductDomain: 'Marketing'

        );

        entities.add(eventSpecEntity);

        entities.add(dataUser);

        entities.add(dataUserAuthentication);

        return EventSpecification(

            event: this,

            contexts: entities

        );

    }

}
````

Here's how you could use the generated code:

```dart
import './lib/tracking/snowplow/snowtype.dart';



const login = Login(method: "google", isSuccess: true);

final eventSpec = login.toUserLogInSpec(

  dataUser: User(

    userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

    firstName: "Ada",

    lastName: "Lovelace",

    organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

    email: "ada@example.com",

    accessLevel: AccessLevel.admin,

  ),

  dataUserAuthentication: UserAuthentication(authType: AuthType.google),

);

await tracker.track(eventSpec.event, contexts: eventSpec.contexts);
```

## Node.js

Snowtype can generate code in TypeScript or JavaScript for the [Node.js tracker](/docs/sources/node-js-tracker/).

### TypeScript

Snowtype generates TypeScript type definitions and typed `track` and `create` functions that import directly from `@snowplow/node-tracker`, plus a `trackXSpec` function, in this case `trackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/node-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```typescript
import { buildSelfDescribingEvent, SelfDescribingJson, Timestamp, Tracker } from '@snowplow/node-tracker';

// Automatically generated by Snowtype



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome

 * of the authentication attempt.

 */

export type Login = {

    /**

     * An optional error code if the login attempt failed (e.g., 'invalid_password',

     * 'user_not_found').

     */

    error_code?: null | string;

    /**

     * Whether the login attempt was successful.

     */

    is_success: boolean;

    /**

     * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

     */

    method: string;

}



/**

 * A subset of the user fields from the users-service

 */

export type User = {

    /**

     * The users access level.

     */

    accessLevel?: AccessLevel;

    /**

     * The users email address.

     */

    email?: null | string;

    /**

     * The users first name.

     */

    firstName: string;

    /**

     * The users job title.

     */

    jobTitle?: null | string;

    /**

     * The users last name.

     */

    lastName: string;

    /**

     * The organization's UUID that the user belongs to.

     */

    organizationId: string;

    /**

     * The users UUID.

     */

    userId: string;

}



/**

 * The users access level.

 */

export type AccessLevel = "Admin" | "User" | "Custom";



/**

 * Data structure that defines how the user logged in

 */

export type UserAuthentication = {

    /**

     * Type of authentication used

     */

    auth_type: AuthType;

}



/**

 * Type of authentication used

 */

export type AuthType = "google" | "facebook" | "manual";



interface CommonEventProperties<T = Record<string, unknown>> {

    /** Add context to an event by setting an Array of Self Describing JSON */

    context?: Array<SelfDescribingJson<T>> | null;

    /** Set the true timestamp or overwrite the device sent timestamp on an event */

    timestamp?: Timestamp | null;

}



/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification: Omit<EventSpecification, 'data_product_domain'> & Partial<Pick<EventSpecification, 'data_product_domain'>>){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}



/**

 * Automatically attached context for event specifications

 */

interface EventSpecification {

    id: string;

    version: number;

    name: string;

    data_product_id: string;

    data_product_name: string;

    data_product_domain?: string;

}



type ContextsOrTimestamp<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribingJson<T>[] | null | undefined }



/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 */

export function trackLogin<T extends {} = any>(tracker: Tracker, login: Login & ContextsOrTimestamp<T>){

    const { context, timestamp, ...data } = login;

    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow Login entity.

 */

export function createLogin(login: Login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 */

export function trackUser<T extends {} = any>(tracker: Tracker, user: User & ContextsOrTimestamp<T>){

    const { context, timestamp, ...data } = user;

    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow User entity.

 */

export function createUser(user: User){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 */

export function trackUserAuthentication<T extends {} = any>(tracker: Tracker, userAuthentication: UserAuthentication & ContextsOrTimestamp<T>){

    const { context, timestamp, ...data } = userAuthentication;

    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow UserAuthentication entity.

 */

export function createUserAuthentication(userAuthentication: UserAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 */

export function trackUserLogInSpec(tracker: Tracker, userLogIn: Login & ContextsOrTimestamp<User | UserAuthentication>){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



    const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn: Login & ContextsOrTimestamp<User | UserAuthentication | EventSpecification> = {

        ...userLogIn,

        context,

    };



    trackLogin(tracker, modifiedUserLogIn);

}
```

Here's how you could use the generated code in a Node.js project:

```tsx
import {

  trackUserLogInSpec,

  createUser,

  createUserAuthentication,

} from "./src/tracking/snowplow";



/*

 * `t` is the tracker instance created by the

 * `tracker` function of @snowplow/node-tracker.

 */



const user = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuth = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec(t, {

  method: "google",

  is_success: true,

  context: [user, userAuth],

});
```

### JavaScript

Snowtype generates JSDoc typedefs and untyped `track` and `create` functions, with a `@typedef {object} Tracker` for the tracker instance parameter, plus a `trackXSpec` function, in this case `trackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/node-tracker",

  "language": "javascript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```javascript
import { buildSelfDescribingEvent } from '@snowplow/node-tracker';

// Automatically generated by Snowtype



/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @typedef {object} Login

 * @property {string} method The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

 * @property {?string} [error_code] An optional error code if the login attempt failed (e.g., 'invalid_password', 'user_not_found').

 * @property {boolean} is_success Whether the login attempt was successful.

 */





/**

 * Typedef for a Login entity context.

 * @typedef {object} LoginContext

 * @property {string} schema The schema of the context.

 * @property { Login } data The data to send for the context.

 */



/**

 * A subset of the user fields from the users-service

 * @typedef {object} User

 * @property {string} userId The users UUID.

 * @property {string} firstName The users first name.

 * @property {string} lastName The users last name.

 * @property {string} organizationId The organization's UUID that the user belongs to.

 * @property {?string} [email] The users email address.

 * @property {?string} [jobTitle] The users job title.

 * @property {"Admin"|"User"|"Custom"|null} [accessLevel] The users access level.

 */





/**

 * Typedef for a User entity context.

 * @typedef {object} UserContext

 * @property {string} schema The schema of the context.

 * @property { User } data The data to send for the context.

 */



/**

 * Data structure that defines how the user logged in

 * @typedef {object} UserAuthentication

 * @property {"google"|"facebook"|"manual"} auth_type Type of authentication used

 */





/**

 * Typedef for a UserAuthentication entity context.

 * @typedef {object} UserAuthenticationContext

 * @property {string} schema The schema of the context.

 * @property { UserAuthentication } data The data to send for the context.

 */



/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}



/**

 * Typedef for a Node.js tracker object.

 * @typedef {object} Tracker

 */



/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @param {Tracker} tracker - Tracker to send the event to.

 * @param { Login & Timestamp & Context } login - Attributes for Login.

 */

export function trackLogin(tracker, login){

    const { context, timestamp, ...data } = login;



    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow Login entity.

 * @param { Login } login - Attributes for Login.

 */

export function createLogin(login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 * @param {Tracker} tracker - Tracker to send the event to.

 * @param { User & Timestamp & Context } user - Attributes for User.

 */

export function trackUser(tracker, user){

    const { context, timestamp, ...data } = user;



    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow User entity.

 * @param { User } user - Attributes for User.

 */

export function createUser(user){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 * @param {Tracker} tracker - Tracker to send the event to.

 * @param { UserAuthentication & Timestamp & Context } userAuthentication - Attributes for UserAuthentication.

 */

export function trackUserAuthentication(tracker, userAuthentication){

    const { context, timestamp, ...data } = userAuthentication;



    tracker.track(buildSelfDescribingEvent({

        event: {

            schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

            data

        }

    }), context, timestamp);

}



/**

 * Creates a Snowplow UserAuthentication entity.

 * @param { UserAuthentication } userAuthentication - Attributes for UserAuthentication.

 */

export function createUserAuthentication(userAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Typedef for UserLogIn event specification contexts.

 * @typedef {object} LoginContexts

 * @property {(UserContext | UserAuthenticationContext)[]} [context] Contexts to include in the UserLogIn event specification.

 */



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 * @param {Tracker} tracker - Tracker to send the tracking event specification to.

 * @param { Login & Timestamp & LoginContexts } userLogIn - Attributes for UserLogIn event specification.

 */

export function trackUserLogInSpec(tracker, userLogIn){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



 const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        context,

    };



    trackLogin(tracker, modifiedUserLogIn);

}
```

Here's how you could use the generated code in a Node.js project:

```javascript
import {

  trackUserLogInSpec,

  createUser,

  createUserAuthentication,

} from "./src/tracking/snowplow";



/*

 * `t` is the tracker instance created by the

 * `tracker` function of @snowplow/node-tracker.

 */



const user = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuth = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec(t, {

  method: "google",

  is_success: true,

  context: [user, userAuth],

});
```

## Golang

Generated code for the [Golang tracker](/docs/sources/golang-tracker/). Snowtype produces Go structs with `mapstructure` tags and `SnowplowFormat()` methods that implement the `SnowplowApplicable` interface, plus a `TrackXSpec` function, in this case `TrackUserLogInSpec`. This function automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-golang-tracker",

  "language": "go",

  "outpath": "./tracking/snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```go
package snowtypegenerated



import "github.com/mitchellh/mapstructure"

import sp "github.com/snowplow/snowplow-golang-tracker/v3/tracker"



// Captured when a user attempts to log in to their account. Tracks the method and outcome

// of the authentication attempt.

type Login struct {

	// An optional error code if the login attempt failed (e.g., 'invalid_password',

	// 'user_not_found').

	ErrorCode                                                                         *string `mapstructure:"error_code"`

	// Whether the login attempt was successful.

	IsSuccess                                                                         bool    `mapstructure:"is_success"`

	// The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

	Method                                                                            string  `mapstructure:"method"`

}



// A subset of the user fields from the users-service

type User struct {

	// The users access level.

	AccessLevel                                         *AccessLevel `mapstructure:"accessLevel,omitempty"`

	// The users email address.

	Email                                               *string      `mapstructure:"email"`

	// The users first name.

	FirstName                                           string       `mapstructure:"firstName"`

	// The users job title.

	JobTitle                                            *string      `mapstructure:"jobTitle"`

	// The users last name.

	LastName                                            string       `mapstructure:"lastName"`

	// The organization's UUID that the user belongs to.

	OrganizationID                                      string       `mapstructure:"organizationId"`

	// The users UUID.

	UserID                                              string       `mapstructure:"userId"`

}



// Data structure that defines how the user logged in

type UserAuthentication struct {

	// Type of authentication used

	AuthType                      AuthType `mapstructure:"auth_type"`

}



// The users access level.

type AccessLevel string



const (

	AccessLevelUser AccessLevel = "User"

	Admin           AccessLevel = "Admin"

	Custom          AccessLevel = "Custom"

)



// Type of authentication used

type AuthType string



const (

	Facebook AuthType = "facebook"

	Google   AuthType = "google"

	Manual   AuthType = "manual"

)



// SnowplowApplicable signifies ability to be formatted as self-describing json

type SnowplowApplicable interface {

	// SnowplowFormat constructs a Snowplow SelfDescribingJson

	SnowplowFormat() (*sp.SelfDescribingJson, error)

}



// internal type to provide functional options for the API

type option func(*sp.SelfDescribingEvent) error



// WithTimestamp returns an option to set the device timestamp

func WithTimestamp(tstamp int64) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.Timestamp = &tstamp

		return nil

	}

}



// WithEventID returns an option to set the event_id

func WithEventID(id string) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.EventId = &id

		return nil

	}

}



// WithTrueTimestamp returns an option to set the true timestamp

func WithTrueTimestamp(tstamp int64) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.TrueTimestamp = &tstamp

		return nil

	}

}



// WithContexts returns an option to set the contexts of an event

func WithContexts(ctx ...SnowplowApplicable) option {

	return func(sde *sp.SelfDescribingEvent) error {

		finalCtx := make([]sp.SelfDescribingJson, len(ctx))

		var ctxErr error

		for i, val := range ctx {

			ctxSdj, ctxErr := val.SnowplowFormat()

			if ctxErr != nil {

				break

			}

			finalCtx[i] = *ctxSdj

		}

		if ctxErr != nil {

			return ctxErr

		}

		sde.Contexts = finalCtx

		return nil

	}

}



// WithSubject returns an option to set the Subject

func WithSubject(subject *sp.Subject) option {

	return func(sde *sp.SelfDescribingEvent) error {

		sde.Subject = subject

		return nil

	}

}



// Entity schema for referencing an event specification

type EventSpecification struct {

	// Identifier for the event specification that the event adheres to

	Id string `mapstructure:"id"`

	Version uint32 `mapstructure:"version"`

	Name string `mapstructure:"name"`

	DataProductId string `mapstructure:"data_product_id"`

	DataProductName string `mapstructure:"data_product_name"`

	DataProductDomain string `mapstructure:"data_product_domain,omitempty"`

}



// SnowplowFormat implements SnowplowApplicable

func (eventSpecification EventSpecification) SnowplowFormat() (*sp.SelfDescribingJson, error) {

	var decodedMap map[string]interface{}

	err := mapstructure.Decode(eventSpecification, &decodedMap)

	if err != nil {

		return nil, err

	}



	sdj := sp.InitSelfDescribingJson("iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4", decodedMap)

	return sdj, nil

}





// SnowplowFormat implements SnowplowApplicable

func (login Login) SnowplowFormat() (*sp.SelfDescribingJson, error) {

	var decodedMap map[string]interface{}

	err := mapstructure.Decode(login, &decodedMap)

	if err != nil {

		return nil, err

	}



	sdj := sp.InitSelfDescribingJson("iglu:com.snplow.msc.aws/login/jsonschema/1-0-0", decodedMap)

	return sdj, nil

}



// TrackLogin tracks a Login

func TrackLogin(tracker *sp.Tracker, login Login, opts ...option) error {

	sdj, err := login.SnowplowFormat()

	if err != nil {

		return err

	}



	sde := &sp.SelfDescribingEvent{Event: sdj}

	for _, opt := range opts {

		opt(sde)

	}



	tracker.TrackSelfDescribingEvent(*sde)

	return nil

}





// SnowplowFormat implements SnowplowApplicable

func (user User) SnowplowFormat() (*sp.SelfDescribingJson, error) {

	var decodedMap map[string]interface{}

	err := mapstructure.Decode(user, &decodedMap)

	if err != nil {

		return nil, err

	}



	sdj := sp.InitSelfDescribingJson("iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1", decodedMap)

	return sdj, nil

}



// TrackUser tracks a User

func TrackUser(tracker *sp.Tracker, user User, opts ...option) error {

	sdj, err := user.SnowplowFormat()

	if err != nil {

		return err

	}



	sde := &sp.SelfDescribingEvent{Event: sdj}

	for _, opt := range opts {

		opt(sde)

	}



	tracker.TrackSelfDescribingEvent(*sde)

	return nil

}





// SnowplowFormat implements SnowplowApplicable

func (userAuthentication UserAuthentication) SnowplowFormat() (*sp.SelfDescribingJson, error) {

	var decodedMap map[string]interface{}

	err := mapstructure.Decode(userAuthentication, &decodedMap)

	if err != nil {

		return nil, err

	}



	sdj := sp.InitSelfDescribingJson("iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0", decodedMap)

	return sdj, nil

}



// TrackUserAuthentication tracks a UserAuthentication

func TrackUserAuthentication(tracker *sp.Tracker, userAuthentication UserAuthentication, opts ...option) error {

	sdj, err := userAuthentication.SnowplowFormat()

	if err != nil {

		return err

	}



	sde := &sp.SelfDescribingEvent{Event: sdj}

	for _, opt := range opts {

		opt(sde)

	}



	tracker.TrackSelfDescribingEvent(*sde)

	return nil

}







// UserLogInApplicable represents a context applicable to UserLogIn event specification

type UserLogInApplicable interface {

	UserLogInFormat() (*sp.SelfDescribingJson, error)

}





// UserLogInFormat implements UserLogInApplicable

func (user User) UserLogInFormat() (*sp.SelfDescribingJson, error) {

	return user.SnowplowFormat()

}





// UserLogInFormat implements UserLogInApplicable

func (userAuthentication UserAuthentication) UserLogInFormat() (*sp.SelfDescribingJson, error) {

	return userAuthentication.SnowplowFormat()

}





// UserLogIn Event Specification

type UserLogIn struct {

	Event    Login

	Contexts []UserLogInApplicable

}





// TrackUserLogInSpec tracks UserLogIn event specification

// ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

func TrackUserLogInSpec(tracker *sp.Tracker, userLogIn UserLogIn, opts ...option) error {

	event := userLogIn.Event

	sdj, err := event.SnowplowFormat()

	if err != nil {

		return err

	}



	sde := &sp.SelfDescribingEvent{Event: sdj}

	var optErr error

	for _, opt := range opts {

		optErr = opt(sde)

		if optErr != nil {

			break

		}

	}

	if optErr != nil {

		return optErr

	}



	eventSpecificationCtx := make([]sp.SelfDescribingJson, len(userLogIn.Contexts))

	var ctxErr error

	for i, val := range userLogIn.Contexts {

		ctxSdj, ctxErr := val.UserLogInFormat()

		if ctxErr != nil {

			break

		}

		eventSpecificationCtx[i] = *ctxSdj

	}

	if ctxErr != nil {

		return ctxErr

	}



	eventSpec := EventSpecification{

		Id: "a965caf1-88a6-4a89-9aea-cc92516a9d56",

		Version: 8,

		Name: "User Log In",

		DataProductId : "57471841-aa79-445d-b4f7-1cbd073a3188",

		DataProductName: "Checkout Flow",

		DataProductDomain: "Marketing",

	}

	eventSpecSdj, err := eventSpec.SnowplowFormat()

	if err != nil {

		return err

	}

	eventSpecificationCtx = append(eventSpecificationCtx, *eventSpecSdj)



	sde.Contexts = append(sde.Contexts, eventSpecificationCtx...)

	tracker.TrackSelfDescribingEvent(*sde)

	return nil

}
```

Here's how you could use the generated code in a Go project:

```go
email := "ada@example.com"

accessLevel := Admin



err := TrackUserLogInSpec(tracker, UserLogIn{

	Event: Login{

		Method:    "google",

		IsSuccess: true,

	},

	Contexts: []UserLogInApplicable{

		User{

			UserID:         "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

			FirstName:      "Ada",

			LastName:       "Lovelace",

			OrganizationID: "57471841-aa79-445d-b4f7-1cbd073a3188",

			Email:          &email,

			AccessLevel:    &accessLevel,

		},

		UserAuthentication{AuthType: Google},

	},

})
```

## Java

Generated code for the [Java tracker](/docs/sources/java-tracker/). Snowtype produces Java classes with a Builder pattern, `toSelfDescribingJson()` methods, and Javadoc comments. Each class is generated as a separate file.

Enums and the `EventSpecification` class are also generated separately, along with a `Snowtype` class containing a static `xSpec()` method, in this case `userLogInSpec()`. This method automatically includes the required entities along with the event specification entity.

```json
{

  "orgId": "your-org-id",

  "tracker": "snowplow-java-tracker",

  "language": "java",

  "outpath": "./src/main/java/snowtype",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;

import com.snowplowanalytics.snowplow.tracker.payload.SelfDescribingJson;



import java.util.HashMap;

import java.util.Map;

import java.util.Objects;





public class Login {

        /** An optional error code if the login attempt failed (e.g., 'invalid_password',

        'user_not_found'). */

        private final String errorCode;

        /** Whether the login attempt was successful. */

        private final boolean isSuccess;

        /** The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'. */

        private final String method;

        private static final String SCHEMA = "iglu:com.snplow.msc.aws/login/jsonschema/1-0-0";



        private Login(Builder builder) {

                this.errorCode = builder.errorCode;

                this.isSuccess = Objects.requireNonNull(builder.isSuccess, "isSuccess is required");

                this.method = Objects.requireNonNull(builder.method, "method is required");

        }



        private Map<String, Object> getPayload() {

                Map<String, Object> payload = new HashMap<>();

                if (errorCode != null) payload.put("error_code", errorCode);

                payload.put("is_success", isSuccess);

                payload.put("method", method);

                return payload;

        }



        /** Creates a self describing json. */

        public SelfDescribingJson toSelfDescribingJson() {

                return new SelfDescribingJson(SCHEMA, getPayload());

        }



        public static class Builder {

                private String errorCode;

                private boolean isSuccess;

                private String method;



                public Builder setErrorCode(String errorCode) {

                        this.errorCode = errorCode;

                        return this;

                }



                public Builder setIsSuccess(boolean isSuccess) {

                        this.isSuccess = isSuccess;

                        return this;

                }



                public Builder setMethod(String method) {

                        this.method = method;

                        return this;

                }



                public Login build() {

                        return new Login(this);

                }

        }

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;

import com.snowplowanalytics.snowplow.tracker.payload.SelfDescribingJson;



import java.util.HashMap;

import java.util.Map;

import java.util.Objects;





public class User {

        /** The users access level. */

        private final AccessLevel accessLevel;

        /** The users email address. */

        private final String email;

        /** The users first name. */

        private final String firstName;

        /** The users job title. */

        private final String jobTitle;

        /** The users last name. */

        private final String lastName;

        /** The organization's UUID that the user belongs to. */

        private final UUID organizationID;

        /** The users UUID. */

        private final UUID userID;

        private static final String SCHEMA = "iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1";



        private User(Builder builder) {

                this.accessLevel = builder.accessLevel;

                this.email = builder.email;

                this.firstName = Objects.requireNonNull(builder.firstName, "firstName is required");

                this.jobTitle = builder.jobTitle;

                this.lastName = Objects.requireNonNull(builder.lastName, "lastName is required");

                this.organizationID = Objects.requireNonNull(builder.organizationID, "organizationID is required");

                this.userID = Objects.requireNonNull(builder.userID, "userID is required");

        }



        private Map<String, Object> getPayload() {

                Map<String, Object> payload = new HashMap<>();

                if (accessLevel != null) payload.put("accessLevel", accessLevel);

                if (email != null) payload.put("email", email);

                payload.put("firstName", firstName);

                if (jobTitle != null) payload.put("jobTitle", jobTitle);

                payload.put("lastName", lastName);

                payload.put("organizationId", organizationID);

                payload.put("userId", userID);

                return payload;

        }



        /** Creates a self describing json. */

        public SelfDescribingJson toSelfDescribingJson() {

                return new SelfDescribingJson(SCHEMA, getPayload());

        }



        public static class Builder {

                private AccessLevel accessLevel;

                private String email;

                private String firstName;

                private String jobTitle;

                private String lastName;

                private UUID organizationID;

                private UUID userID;



                public Builder setAccessLevel(AccessLevel accessLevel) {

                        this.accessLevel = accessLevel;

                        return this;

                }



                public Builder setEmail(String email) {

                        this.email = email;

                        return this;

                }



                public Builder setFirstName(String firstName) {

                        this.firstName = firstName;

                        return this;

                }



                public Builder setJobTitle(String jobTitle) {

                        this.jobTitle = jobTitle;

                        return this;

                }



                public Builder setLastName(String lastName) {

                        this.lastName = lastName;

                        return this;

                }



                public Builder setOrganizationID(UUID organizationID) {

                        this.organizationID = organizationID;

                        return this;

                }



                public Builder setUserID(UUID userID) {

                        this.userID = userID;

                        return this;

                }



                public User build() {

                        return new User(this);

                }

        }

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;

import com.snowplowanalytics.snowplow.tracker.payload.SelfDescribingJson;



import java.util.HashMap;

import java.util.Map;

import java.util.Objects;





public class UserAuthentication {

        /** Type of authentication used */

        private final AuthType authType;

        private static final String SCHEMA = "iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0";



        private UserAuthentication(Builder builder) {

                this.authType = Objects.requireNonNull(builder.authType, "authType is required");

        }



        private Map<String, Object> getPayload() {

                Map<String, Object> payload = new HashMap<>();

                payload.put("auth_type", authType.getString());

                return payload;

        }



        /** Creates a self describing json. */

        public SelfDescribingJson toSelfDescribingJson() {

                return new SelfDescribingJson(SCHEMA, getPayload());

        }



        public static class Builder {

                private AuthType authType;



                public Builder setAuthType(AuthType authType) {

                        this.authType = authType;

                        return this;

                }



                public UserAuthentication build() {

                        return new UserAuthentication(this);

                }

        }

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;



/**

 * The users access level.

*/

public enum AccessLevel{

        ADMIN("Admin"),

        CUSTOM("Custom"),

        USER("User");



        private final String value;



        AccessLevel(String value) {

                this.value = value;

        }



        public String getString() {

                return value;

        }

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;



/**

 * Type of authentication used

*/

public enum AuthType{

        FACEBOOK("facebook"),

        GOOGLE("google"),

        MANUAL("manual");



        private final String value;



        AuthType(String value) {

                this.value = value;

        }



        public String getString() {

                return value;

        }

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;

import com.snowplowanalytics.snowplow.tracker.payload.SelfDescribingJson;



import java.util.HashMap;

import java.util.Map;





public class EventSpecification {

        private final String id;

        private final String name;

        private final int version;

        private final String dataProductId;

        private final String dataProductName;

        private final String dataProductDomain;

        private static final String SCHEMA =

                "iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4";



        public EventSpecification(String id, int version, String name,

                        String dataProductId, String dataProductName, String dataProductDomain) {

                this.id = id;

                this.version = version;

                this.name = name;

                this.dataProductId = dataProductId;

                this.dataProductName = dataProductName;

                this.dataProductDomain = dataProductDomain;

        }



        public EventSpecification(String id, int version, String name,

                        String dataProductId, String dataProductName) {

                this(id, version, name, dataProductId, dataProductName, null);

        }



        private Map<String, Object> getPayload() {

                Map<String, Object> payload = new HashMap<>();

                payload.put("id", id);

                payload.put("version", version);

                payload.put("name", name);

                payload.put("data_product_id", dataProductId);

                payload.put("data_product_name", dataProductName);

                if (dataProductDomain != null) {

                        payload.put("data_product_domain", dataProductDomain);

                }

                return payload;

        }



        /** Creates an entity that can be added to events. */

        public SelfDescribingJson toSelfDescribingJson() {

                return new SelfDescribingJson(SCHEMA, getPayload());

        }

}
```

```java
package com.snowplowanalytics.snowplow.snowtype;

import com.snowplowanalytics.snowplow.tracker.payload.SelfDescribingJson;

import com.snowplowanalytics.snowplow.tracker.events.SelfDescribing;



import java.util.Arrays;



public class Snowtype {



        public static SelfDescribing userLogInSpec(Login login,

                        User entityUser, UserAuthentication entityUserAuthentication) {

                SelfDescribingJson event = login.toSelfDescribingJson();

                EventSpecification eventSpec = new EventSpecification(

                        "a965caf1-88a6-4a89-9aea-cc92516a9d56",

                        8,

                        "User Log In",

                        "57471841-aa79-445d-b4f7-1cbd073a3188",

                        "Checkout Flow",

                        "Marketing"

                );

                return SelfDescribing

                        .builder()

                        .eventData(event)

                        .customContext(Arrays.asList(

                                entityUser.toSelfDescribingJson(),

                                entityUserAuthentication.toSelfDescribingJson(),

                                eventSpec.toSelfDescribingJson()))

                        .build();

        }

}
```

Here's how you could use the generated code:

```java
import com.snowplowanalytics.snowplow.snowtype.*;



Login login = new Login.Builder()

    .setIsSuccess(true)

    .setMethod("google")

    .build();



User user = new User.Builder()

    .setFirstName("Ada")

    .setLastName("Lovelace")

    .setOrganizationID(UUID.fromString("57471841-aa79-445d-b4f7-1cbd073a3188"))

    .setUserID(UUID.fromString("a1b2c3d4-e5f6-7890-abcd-ef1234567890"))

    .setEmail("ada@example.com")

    .setAccessLevel(AccessLevel.ADMIN)

    .build();



UserAuthentication userAuth = new UserAuthentication.Builder()

    .setAuthType(AuthType.GOOGLE)

    .build();



tracker.track(Snowtype.userLogInSpec(login, user, userAuth));
```

## Google Tag Manager (JavaScript)

Generated code for [Google Tag Manager](/docs/sources/google-tag-manager/). Snowtype produces a single function that assigns tracking methods to `window.__snowtype`, including `trackXSpec`, designed to be used in a GTM [Custom JavaScript Variable](/docs/event-studio/implement-tracking/working-with-gtm/).

Snowtype will also generate a minified version.

```json
{

  "orgId": "your-org-id",

  "tracker": "google-tag-manager",

  "language": "javascript-gtm",

  "outpath": "./gtm",

  "dataProductIds": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"],

  "dataStructures": [],

  "igluCentralSchemas": []

}
```

```javascript
// Automatically generated by Snowtype

// This code should be copied and pasted in a Google Tag Manager Custom JavaScript Variable



function snowtypeInit() {

    function _objectWithoutProperties(e, t) { if (null == e) return {}; var o, r, i = _objectWithoutPropertiesLoose(e, t); if (Object.getOwnPropertySymbols) { var n = Object.getOwnPropertySymbols(e); for (r = 0; r < n.length; r++) o = n[r], t.indexOf(o) >= 0 || {}.propertyIsEnumerable.call(e, o) && (i[o] = e[o]); } return i; }

    function _objectWithoutPropertiesLoose(r, e) { if (null == r) return {}; var t = {}; for (var n in r) if ({}.hasOwnProperty.call(r, n)) { if (e.indexOf(n) >= 0) continue; t[n] = r[n]; } return t; }



    /**

    * Creates a Snowplow Event Specification entity.

    */

    function createEventSpecification(eventSpecification){

        return {

            schema: 'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

            data: eventSpecification,

        }

    }



    window.__snowtype = {

        /**

        * Track a Snowplow event for Login.

        * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

        * @param { Login & Timestamp & Context } login - Attributes for Login.

        * @param {string[]} [trackers] - Tracker names to send the event to.

        */

        trackLogin: function(login, trackers){

                var trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

                var context = login.context;

                var timestamp = login.timestamp;

                var data = _objectWithoutProperties(login, ["context", "timestamp"]);

                window.snowplow('trackSelfDescribingEvent' + trackerNames, {

                        event: {

                                schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

                                data: data

                        },

                        context: context,

                        timestamp: timestamp,

                });

        },

        /**

        * Creates a Snowplow Login entity.

        * @param { Login } login - Attributes for Login.

        */

        createLogin: function(login){

                return {

                        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

                        data: login

                };

        },

        /**

        * Track a Snowplow event for User.

        * A subset of the user fields from the users-service

        * @param { User & Timestamp & Context } user - Attributes for User.

        * @param {string[]} [trackers] - Tracker names to send the event to.

        */

        trackUser: function(user, trackers){

                var trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

                var context = user.context;

                var timestamp = user.timestamp;

                var data = _objectWithoutProperties(user, ["context", "timestamp"]);

                window.snowplow('trackSelfDescribingEvent' + trackerNames, {

                        event: {

                                schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

                                data: data

                        },

                        context: context,

                        timestamp: timestamp,

                });

        },

        /**

        * Creates a Snowplow User entity.

        * @param { User } user - Attributes for User.

        */

        createUser: function(user){

                return {

                        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

                        data: user

                };

        },

        /**

        * Track a Snowplow event for UserAuthentication.

        * Data structure that defines how the user logged in

        * @param { UserAuthentication & Timestamp & Context } userAuthentication - Attributes for UserAuthentication.

        * @param {string[]} [trackers] - Tracker names to send the event to.

        */

        trackUserAuthentication: function(userAuthentication, trackers){

                var trackerNames = (trackers && trackers.length) ? ':' + trackers.join(';') : '';

                var context = userAuthentication.context;

                var timestamp = userAuthentication.timestamp;

                var data = _objectWithoutProperties(userAuthentication, ["context", "timestamp"]);

                window.snowplow('trackSelfDescribingEvent' + trackerNames, {

                        event: {

                                schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

                                data: data

                        },

                        context: context,

                        timestamp: timestamp,

                });

        },

        /**

        * Creates a Snowplow UserAuthentication entity.

        * @param { UserAuthentication } userAuthentication - Attributes for UserAuthentication.

        */

        createUserAuthentication: function(userAuthentication){

                return {

                        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

                        data: userAuthentication

                };

        },

        /**

        * Tracks a UserLogIn event specification.

        * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

        * @param { Login & Timestamp & UserLogInContexts } userLogIn - Attributes for UserLogIn event specification.

        * @param {string[]} [trackers] - Tracker names to send the event specification to.

        */

        trackUserLogInSpec: function(userLogIn, trackers){

                var eventSpecificationContext = createEventSpecification({

                        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

                        version: 8,

                        name: 'User Log In',

                        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

                        data_product_name: 'Checkout Flow',

                        data_product_domain: 'Marketing'

                });



                var context = Array.isArray(userLogIn.context)

                        ? userLogIn.context.concat(eventSpecificationContext)

                        : [eventSpecificationContext];



                var modifiedUserLogIn = Object.assign(

                        userLogIn,

                        { context: context }

                );



                this.trackLogin(modifiedUserLogIn, trackers);

        },

     };

     return undefined;

 }



/**

 * Typedef for a UserLogInContexts

 * @typedef {object} UserLogInContexts

 * @property {(UserContext | UserAuthenticationContext)[]} [context] Contexts to include in the UserLogIn event specification.

 */





/**

 * Typedef for a DeviceTimestamp

 * @typedef {object} DeviceTimestamp

 * @property {'dtm'} type The value of 'dtm'

 * @property {number} value The value of the device timestamp

 */



/**

 * Typedef for a TrueTimestamp

 * @typedef {object} TrueTimestamp

 * @property {'ttm'} type The value of 'ttm'

 * @property {number} value The value of the true timestamp

 */



/**

 * Typedef for a Timestamp

 * @typedef {object} Timestamp

 * @property {number|TrueTimestamp|DeviceTimestamp} [timestamp] The value of the timestamp

 */



/**

 * Typedef for a SelfDescribingJson

 * @typedef {object} SelfDescribingJson

 * @property {string} schema The schema of the context

 * @property {object} data The data to send for the context

 */



/**

 * Typedef for a Context

 * @typedef {object} Context

 * @property {SelfDescribingJson[]} [context] Contexts to include in the event

 */



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 * @typedef {object} Login

 * @property {string} method The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

 * @property {?string} [error_code] An optional error code if the login attempt failed (e.g., 'invalid_password', 'user_not_found').

 * @property {boolean} is_success Whether the login attempt was successful.

 */





/**

 * Typedef for a Login entity context.

 * @typedef {object} LoginContext

 * @property {string} schema The schema of the context.

 * @property { Login } data The data to send for the context.

 */



/**

 * A subset of the user fields from the users-service

 * @typedef {object} User

 * @property {string} userId The users UUID.

 * @property {string} firstName The users first name.

 * @property {string} lastName The users last name.

 * @property {string} organizationId The organization's UUID that the user belongs to.

 * @property {?string} [email] The users email address.

 * @property {?string} [jobTitle] The users job title.

 * @property {"Admin"|"User"|"Custom"|null} [accessLevel] The users access level.

 */





/**

 * Typedef for a User entity context.

 * @typedef {object} UserContext

 * @property {string} schema The schema of the context.

 * @property { User } data The data to send for the context.

 */



/**

 * Data structure that defines how the user logged in

 * @typedef {object} UserAuthentication

 * @property {"google"|"facebook"|"manual"} auth_type Type of authentication used

 */





/**

 * Typedef for a UserAuthentication entity context.

 * @typedef {object} UserAuthenticationContext

 * @property {string} schema The schema of the context.

 * @property { UserAuthentication } data The data to send for the context.

 */
```

---

# Generate tracking code
> Use Snowtype to generate type-safe tracking code from event specifications, tracking plans, data structures, and Iglu Central schemas.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/generate-tracking-code/

Once you've [installed and initialized Snowtype](/docs/event-studio/implement-tracking/install-snowtype/), you can generate tracking code from the [sources you configured](/docs/event-studio/implement-tracking/snowtype-config/).

To generate tracking code, run:

```bash
npx snowtype generate
```

When you run `snowtype generate`, Snowtype will produce a single output file, at the path set in your configuration file. The contents will vary based on the tracker and language you selected, but generally include:

- Types, interfaces, or classes for each schema
- Functions to track the schemas as self-describing events or entities
- For event specifications, functions to track the events with the correct schemas and instructions defined in the specification

Snowtype will generate code for you to track all schemas as either a self-describing event or entity, regardless of how you've defined your data structures in Console.

The first time you run `generate`, Snowtype creates a `.snowtype-lock.json` file next to your configuration file. This pins the event specification versions used for generation, so subsequent runs produce consistent output. To check for newer event specification versions, [use `snowtype update`](/docs/event-studio/implement-tracking/keep-code-up-to-date/).

The generated code isn't minified and includes inline documentation. You can modify it to suit your project, but any changes will be overwritten the next time you run `generate`.

The code expects the relevant [Snowplow tracker](/docs/sources/) to already be installed in your project. Snowtype doesn't install trackers for you.

## Data structures

For data structures and Iglu Central schemas, Snowtype generates two functions: one to track the schema as a self-describing event, and one to create an entity object. It will also create whatever types or helper functions are available for the selected language.

See the [Use the generated code](#use-the-generated-code) section below for an example of how to use the generated functions in your code.

This example shows generated Browser tracker TypeScript code for the `web_page` schema from Iglu Central, and a custom `product` data structure. The inline comments are generated from the schema descriptions.

To see the output for different trackers and languages, check out the [full examples page](/docs/event-studio/implement-tracking/generate-tracking-code/example-data-structures/).

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

```tsx
import { trackSelfDescribingEvent, CommonEventProperties, SelfDescribingJson } from '@snowplow/browser-tracker';

// Automatically generated by Snowtype



/**

 * Schema for a web page

 */

export type WebPage = {

    /**

     * The page view ID.

     */

    id: string;

}



/**

 * Schema for a product entity

 */

export type Product = {

    /**

     * The product category.

     */

    category: string;

    /**

     * The currency the product is listed in.

     */

    currency: string;

    /**

     * The product ID (SKU).

     */

    id: string;

    /**

     * The product name.

     */

    name: string;

    /**

     * The price of the product.

     */

    price: number;

}



type ContextsOrTimestamp<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribingJson<T>[] | null | undefined }

/**

 * Track a Snowplow event for WebPage.

 * Schema for a web page

 */

export function trackWebPage<T extends {} = any>(webPage: WebPage & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = webPage;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow WebPage entity.

 */

export function createWebPage(webPage: WebPage){

    return {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: webPage

    }

}

/**

 * Track a Snowplow event for Product.

 * Schema for a product entity

 */

export function trackProduct<T extends {} = any>(product: Product & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = product;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow Product entity.

 */

export function createProduct(product: Product){

    return {

        schema: 'iglu:com.example/product/jsonschema/1-0-0',

        data: product

    }

}
```

## Event specifications

For event specifications, which have more requirements and metadata than data structures or Iglu Central schemas, Snowtype generates additional code.

Along with the self-describing event and entity functions for each schema, it also produces a function to track the event according to the instructions defined in the specification. This includes any required entities, as well as an event specification entity that links this event to its tracking plan.

See the [Use the generated code](#use-the-generated-code) section below for an example of how to use the generated functions in your code. For JavaScript and TypeScript, the key function is called `track{EventSpecificationName}Spec`.

> **Note:** Changes to event specifications increases the version number. Your [lockfile](/docs/event-studio/implement-tracking/keep-code-up-to-date/#keep-event-specifications-up-to-date) pins specific event specification versions.
>
> If you don't see the expected changes in your generated code after editing the event specification, try running `snowtype update`.

This example shows generated Browser tracker TypeScript code for a `User Log In` event specification. The event specification has a `login` event data structure, plus two entities: `user_authentication` and `user`.

To see the output for different trackers and languages, check out the [full examples page](/docs/event-studio/implement-tracking/generate-tracking-code/example-event-specs/).

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "igluCentralSchemas": [],

  "dataStructures": [],

  "eventSpecificationIds": ["a965caf1-88a6-4a89-9aea-cc92516a9d56"]

}
```

```tsx
import { trackSelfDescribingEvent, CommonEventProperties, SelfDescribingJson } from '@snowplow/browser-tracker';

// Automatically generated by Snowtype



/**

 * Captured when a user attempts to log in to their account. Tracks the method and outcome

 * of the authentication attempt.

 */

export type Login = {

    /**

     * An optional error code if the login attempt failed (e.g., 'invalid_password',

     * 'user_not_found').

     */

    error_code?: null | string;

    /**

     * Whether the login attempt was successful.

     */

    is_success: boolean;

    /**

     * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

     */

    method: string;

}



/**

 * A subset of the user fields from the users-service

 */

export type User = {

    /**

     * The users access level.

     */

    accessLevel?: AccessLevel;

    /**

     * The users email address.

     */

    email?: null | string;

    /**

     * The users first name.

     */

    firstName: string;

    /**

     * The users job title.

     */

    jobTitle?: null | string;

    /**

     * The users last name.

     */

    lastName: string;

    /**

     * The organization's UUID that the user belongs to.

     */

    organizationId: string;

    /**

     * The users UUID.

     */

    userId: string;

}



/**

 * The users access level.

 */

export type AccessLevel = "Admin" | "User" | "Custom";



/**

 * Data structure that defines how the user logged in

 */

export type UserAuthentication = {

    /**

     * Type of authentication used

     */

    auth_type: AuthType;

}



/**

 * Type of authentication used

 */

export type AuthType = "google" | "facebook" | "manual";



/**

 * Creates a Snowplow Event Specification entity.

 */

export function createEventSpecification(eventSpecification: Omit<EventSpecification, 'data_product_domain'> & Partial<Pick<EventSpecification, 'data_product_domain'>>){

    return {

        schema:

            'iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-4',

        data: eventSpecification,

    }

}



/**

 * Automatically attached context for event specifications

 */

interface EventSpecification {

    id: string;

    version: number;

    name: string;

    data_product_id: string;

    data_product_name: string;

    data_product_domain?: string;

}



type ContextsOrTimestamp<T = any> = Omit<CommonEventProperties<T>, 'context'> & { context?: SelfDescribingJson<T>[] | null | undefined }

/**

 * Track a Snowplow event for Login.

 * Captured when a user attempts to log in to their account. Tracks the method and outcome of the authentication attempt.

 */

export function trackLogin<T extends {} = any>(login: Login & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = login;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow Login entity.

 */

export function createLogin(login: Login){

    return {

        schema: 'iglu:com.snplow.msc.aws/login/jsonschema/1-0-0',

        data: login

    }

}

/**

 * Track a Snowplow event for User.

 * A subset of the user fields from the users-service

 */

export function trackUser<T extends {} = any>(user: User & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = user;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow User entity.

 */

export function createUser(user: User){

    return {

        schema: 'iglu:com.snowplowanalytics.console/user/jsonschema/1-0-1',

        data: user

    }

}

/**

 * Track a Snowplow event for UserAuthentication.

 * Data structure that defines how the user logged in

 */

export function trackUserAuthentication<T extends {} = any>(userAuthentication: UserAuthentication & ContextsOrTimestamp<T>, trackers?: string[]){

    const { context, timestamp, ...data } = userAuthentication;

    const event: SelfDescribingJson = {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data

    };



    trackSelfDescribingEvent({

        event,

        context,

        timestamp,

    }, trackers);

}



/**

 * Creates a Snowplow UserAuthentication entity.

 */

export function createUserAuthentication(userAuthentication: UserAuthentication){

    return {

        schema: 'iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0',

        data: userAuthentication

    }

}



/**

 * Tracks a UserLogIn event specification.

 * ID: a965caf1-88a6-4a89-9aea-cc92516a9d56

 */

export function trackUserLogInSpec(userLogIn: Login & ContextsOrTimestamp<User | UserAuthentication>, trackers?: string[]){

    const eventSpecificationContext = createEventSpecification({

        id: 'a965caf1-88a6-4a89-9aea-cc92516a9d56',

        version: 8,

        name: 'User Log In',

        data_product_id: '57471841-aa79-445d-b4f7-1cbd073a3188',

        data_product_name: 'Checkout Flow',

        data_product_domain: 'Marketing'

    });



    const context = Array.isArray(userLogIn.context)

        ? [...userLogIn.context, eventSpecificationContext]

        : [eventSpecificationContext];



    const modifiedUserLogIn = {

        ...userLogIn,

        context,

    };



    trackLogin<User | UserAuthentication | EventSpecification>(modifiedUserLogIn, trackers);

}
```

### Event specification entity

Code generated for event specifications includes an entity that allows you to link the tracked event to its event specification and tracking plan.

### `event_specification`

**Type:** Entity

Entity schema for referencing an event specification

**Properties:**

| Property                       | Description                                                                         |
| ------------------------------ | ----------------------------------------------------------------------------------- |
| `id` _string_                  | _Required._ Identifier for the event specification that the event adheres to        |
| `name` _string_                | _Optional._ Name for the event specification that the event adheres to              |
| `data_product_id` _string_     | _Optional._ Identifier for the data product that the event specification belongs to |
| `data_product_name` _string_   | _Optional._ Name for the data product that the event specification belongs to       |
| `data_product_domain` _string_ | _Optional._ Domain for the data product that the event specification belongs to     |
| `version` _integer_            | _Optional._ Version of the event specification that the event adheres to            |

## Property rules and instructions

Snowtype will take into account property rules that you define within your event specifications.

For example, if you add an instruction that a property must have one of a specific set of values, Snowtype will generate a type for that property that only allows those values.

If you include an instruction that's a detailed explanation of how to implement the data, this will be included as a comment in the generated code. It'll replace the description from the underlying data structure.

Snowtype supports entity cardinality rules for the [Browser tracker only](/docs/event-studio/implement-tracking/client-side-validation/).

> **Note:** Changes to instructions increase the event specification version number. Your [lockfile](/docs/event-studio/implement-tracking/keep-code-up-to-date/#keep-event-specifications-up-to-date) pins specific event specification versions.
>
> If you don't see the expected updates in your generated code after making changes, try running `snowtype update`.

This example uses the `User Log In` event specification from the previous section. Within the `Login` event data structure, property rules have been set for two of the properties:

- `error_code`: a detailed instruction explaining what values to supply
- `method`: an enum rule limiting the allowed values to `email`, `google`, `apple`, and `saml`

In the generated output, the instruction on `error_code` appears as an inline comment, and `method` becomes a union type instead of `string`:

```tsx
// Before instructions



export type Login = {

    /**

     * An optional error code if the login attempt failed (e.g., 'invalid_password',

     * 'user_not_found').

     */

    error_code?: null | string;

    /**

     * Whether the login attempt was successful.

     */

    is_success: boolean;

    /**

     * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

     */

    method: string;

}



// After instructions



export type Login = {

    /**

     * Provide the raw error code from the authentication service, for example:

     * 'AUTH_001' for invalid credentials, 'AUTH_002' for an expired session,

     * or 'AUTH_003' for a locked account. Leave null if the login succeeded.

     */

    error_code?: null | string;

    /**

     * Whether the login attempt was successful.

     */

    is_success: boolean;

    /**

     * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

     */

    method: LoginMethod;

}



/**

 * The method used to authenticate, such as 'email', 'google', 'apple', or 'saml'.

 */

export type LoginMethod = "email" | "google" | "apple" | "saml";
```

### Markdown instructions

When generating code for event specifications, you can also produce a Markdown file containing the implementation instructions defined for each specification. This can be useful for sharing implementation requirements with developers who may not have access to Console.

Use the `--instructions` flag:

```bash
npx snowtype generate --instructions
```

The generated Markdown includes:

- Trigger descriptions
- Implementation rules
- Images uploaded on your event specification triggers
- App identifiers and URLs where the event should fire
- Links to the generated code for each event specification

If you generate code for multiple event specifications, the instructions for each will be included in the same file, separated by headings.

This example shows the generated instruction file for a `User Log In` event specification:

```markdown
# Implementation instructions



_A guide on how to implement tracking for the generated Event Specifications._



**Table of contents:**

- [User Log In](#user-log-in)





## [User Log In](https://console.snowplowanalytics.com/177234df-d425-412e-ad8d-8b97515b2807/data-products/57471841-aa79-445d-b4f7-1cbd073a3188/event-specifications/a965caf1-88a6-4a89-9aea-cc92516a9d56)



|       |  |



| ----------- | ----------- |

| **Id** | a965caf1-88a6-4a89-9aea-cc92516a9d56 |

| **Version** | 8 |

| **Data Product Id** | 57471841-aa79-445d-b4f7-1cbd073a3188 |

| **Source Application/s** | None selected |

| **Event Data Structure** | login/1-0-0 |

| **Entity Data Structures** | user/1-0-1, user_authentication/1-0-0 |

| **Code** | [Link](./snowplow.ts#L185) |

| **Data Product Domain** | Marketing |







#### Entity Cardinality Rules

|    Name   | Required  | Number of entities  |



| ----------- | ----------- |  ----------- |

user | ❌ | Between `0` and `1`

user_authentication | ✅ | Exactly `1`







### Trigger



#### Screenshot

_N/A_



#### In which application(s) does this trigger apply?



_N/A_



#### The URL of the page on which this event specification triggers:



_N/A_



#### Notes



Track this when the user clicks the Log in button on the log in form/selection modal
```

## Prevent generation from development schemas

When you are developing new data structures, they may only be deployed to your development environment. Generating and using tracking code from development-only schemas and shipping it to production can cause [failed events](/docs/fundamentals/failed-events/), because the production pipeline doesn't have those schemas.

By default, Snowtype prints a warning when it detects development-only schemas:

```bash
✔: Valid configuration found

⠦ Generating...

 ⚠ Warning: Generating code using schemas in DEV environment, tracking against those will result in failed events. Schemas in DEV:

iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0



✔: Code generated, writing...
```

To turn this into a hard error that stops generation, use the `--disallowDevSchemas` flag:

```bash
npx snowtype generate --disallowDevSchemas
```

```bash
✔: Valid configuration found

⠼ Generating...

 ⚠ Warning: Generating code using schemas in DEV environment, tracking against those will result in failed events. Schemas in DEV:

iglu:com.snplow.msc.aws/user_authentication/jsonschema/1-0-0



✗ Error: Code generation using schemas in DEV environment not allowed. Publish the schemas and retry.
```

This is especially useful in CI/CD pipelines where you want to catch the problem before code reaches production.

## Use the generated code

How to implement the generated code depends on the tracker and language you're using.

This example demonstrates how to use Snowtype's generated code for the Browser tracker, based on the generated code shown above.

Check out the full [data structures](/docs/event-studio/implement-tracking/generate-tracking-code/example-data-structures/) and [event specifications](/docs/event-studio/implement-tracking/generate-tracking-code/example-event-specs/) example pages for different languages and trackers.

Using the [generated data structure and Iglu Central schema code](#data-structures) from the previous section:

```typescript
import {

  trackWebPage,

  createProduct,

  WebPage,

  Product,

  createWebPage,

} from "./{outpath}/snowplow";



/* Track a self-describing event */

trackWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });



/* Track an event with an entity attached */

const productEntity = createProduct({

  id: "Product id",

  name: "Snowplow product",

  currency: "EUR",

  price: 10,

  category: "Snowplow/Shoes",

});

trackWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [productEntity],

});



/* Enforce specific entity types using type arguments */

const webPageEntity = createWebPage({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

});

trackWebPage<Product | WebPage>({

  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",

  context: [productEntity, webPageEntity],

});
```

Using the [generated event specification code](#event-specifications) from the previous section:

```typescript
import {

  trackUserLogInSpec,

  createUser,

  createUserAuthentication,

} from "./src/tracking/snowplow";



const userEntity = createUser({

  userId: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",

  firstName: "Ada",

  lastName: "Lovelace",

  organizationId: "57471841-aa79-445d-b4f7-1cbd073a3188",

  email: "ada@example.com",

  accessLevel: "Admin",

});



const userAuthEntity = createUserAuthentication({ auth_type: "google" });



trackUserLogInSpec({

  method: "google",

  is_success: true,

  context: [userEntity, userAuthEntity],

});
```

## Deprecation and hidden warnings

When you `generate` code from event specifications, Snowtype checks whether the data structures referenced by each specification are up to date. If a newer version of a data structure exists, or if the data structure has been hidden in Console, Snowtype adds a warning comment to the generated tracking function.

There are two types of warnings:

- Outdated data structure: the event specification references a data structure version that isn't the latest. Snowtype marks the generated function with `@deprecated Outdated data structure detected.` in JavaScript/TypeScript, or an equivalent comment in other languages.
- Hidden data structure: the event specification references a data structure that has been hidden in Console. Snowtype marks the generated function with `Warning: Hidden data structure detected.`

If both conditions apply, both warnings appear together:

```typescript
/**

 * @deprecated Outdated data structure detected.

 * Warning: Hidden data structure detected.

 * Tracks a MakesPurchase event specification.

 * ID: 132a582b-88f3-4bcc-b1d5-c89e5813bbc9

 */

export function trackMakesPurchaseSpec(

    ...

)
```

These warnings surface in your IDE so you can see which event specifications need attention.

### Show warnings only for production-deployed updates

By default, Snowtype flags outdated data structures in event specifications, regardless of where the newer version is deployed. If you only want to see deprecation warnings when a newer version has been deployed to your production environment, use the `--deprecateOnlyOnProdAvailableUpdates` flag:

```bash
npx snowtype generate --deprecateOnlyOnProdAvailableUpdates
```

This suppresses outdated warnings when the newer schema version only exists in development, which can reduce noise during active development. You can also set this in your [configuration file](/docs/event-studio/implement-tracking/configuration-reference/#command-options).

## Name collisions and reusing objects

Snowtype has built-in logic to handle multiple uses of the same data structures, and limited support for handling name collisions.

### Multiple versions of the same schema

If you generate code for multiple versions of the same data structure or Iglu Central schema, Snowtype will automatically disambiguate them by appending numbers to the types and function names.

For example, if you provide `iglu:com.example/message_delayed/jsonschema/2-0-0` and `iglu:com.example/message_delayed/jsonschema/3-0-0` in your configuration, the generated code will include e.g. `type MessageDelayed` (no number for the first instance) and `type MessageDelayed1` (number appended for subsequent instances).

The numbering is based on the order of the sources in your configuration file. Snowtype processes local repository data structures first, then Iglu Central schemas, then Console data structures.

### Same data structure in multiple event specifications

If you use the same data structure in multiple event specifications, Snowtype will generate only one set of functions for that data structure. You can use them wherever they're needed.

If you use the same data structure with different instructions or property rules in different event specifications, Snowtype will automatically disambiguate them based on the name of the event specification.

For example, you may have two event specifications, `Add To Cart` and `User Log In`, using the same `user` entity data structure with different instructions. Snowtype will generate code that includes e.g. `type User` (no suffix for the first instance) and `type UserUserLogIn` (event specification name appended for subsequent instances).

Snowtype will always add the name suffix for a data structure that has instructions, even if you've removed the instructions from it.

The autogenerated inline documentation comments and types will tell you which functions to use.

### Event specifications with the same name

If you have multiple event specifications with the same name, you might experience name collisions, depending on which tracker you're using.

> **Tip:** We recommend giving each event specification a unique name.

In Kotlin, Swift, and Dart, Snowtype uses the tracking plan name for extensions, which prevents name collisions.

For example, you may have two event specifications with the same name, `User Log In`, in two different tracking plans, `Checkout Flow` and `User Management`. In the `Checkout Flow` tracking plan, the `User Log In` event specification has the event data structure `login`. In the `User Management` tracking plan, the `User Log In` event specification has the event data structure `authentication_attempt`.

When you generate code for these specifications in Kotlin or Swift, Snowtype produces two separate functions to track each specification, with different extension functions based on the tracking plan name:

**Kotlin:**

```kotlin
fun Login.toUserLogInSpec() // for Checkout Flow

fun AuthenticationAttempt.toUserLogInSpec() // for User Management
```

**Swift:**

```swift
extension Login {

    func toUserLogInSpec() // for Checkout Flow

}



extension AuthenticationAttempt {

    func toUserLogInSpec() // for User Management

}
```

***

In other languages, the generated function names are based only on the event specification name, which can lead to collisions if you have multiple specifications with the same name.

```typescript
// For Checkout Flow

export function trackUserLogInSpec(userLogIn: Login & ContextsOrTimestamp, trackers?: string[])



// For User Management — duplicate identifier causes SyntaxError

export function trackUserLogInSpec(userLogIn: AuthenticationAttempt & ContextsOrTimestamp, trackers?: string[])
```

## Snowtype limitations

Snowtype **does not work** with [tracking plan templates](/docs/event-studio/tracking-plans/templates/). Track these manually using the standard tracker API.

### Excluded schemas

Snowtype doesn't generate code for certain out-of-the-box Snowplow event schemas. Use the standard tracker API to track the event data structure as a self-describing event, or use the out-of-the-box tracking functionality.

If your event specification uses any of these schemas for the event data structure, Snowtype will generate code for the event specification and entities only:

- `iglu:com.snowplowanalytics.snowplow/page_ping/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`
- `iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/expose_element/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/create_element/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/custom_event/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/application_background/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.snowplow/application_foreground/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.mobile/screen_view/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.mobile/screen_end/jsonschema/1-0-0`
- `iglu:com.snowplowanalytics.mobile/application_install/jsonschema/1-0-0`

You'll see a warning logged when you generate code using these: `Warning: Currently code will not be generated for Standard events. Skipping`.

---

# Implement tracking with Snowtype
> Generate and implement tracking code from your event specifications using Snowtype, Console code snippets, or manual SDK integration.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/

Once you've defined your [tracking plans](/docs/event-studio/tracking-plans/) and [event specifications](/docs/event-studio/tracking-plans/event-specifications/), the next step is implementing the tracking code in your applications.

The best way to implement tracking is to use our CLI code generation tool, **Snowtype**. Snowtype reads your [event specifications](/docs/event-studio/tracking-plans/event-specifications/) and [data structures](/docs/event-studio/data-structures/) from your Snowplow account, and generates type-safe tracking code for the language and tracker you specify.

This provides several key advantages over manual implementation:

- Quicker implementation: reduce the work needed to produce production-ready tracking code
- Type safety: ensure your tracking code is consistent with your schemas and catch errors before they reach your pipeline
- Workflow integration: use CI/CD GitOps-like processes to [keep your tracking code in sync](/docs/event-studio/implement-tracking/keep-code-up-to-date/) with your schemas

For custom tracking on web, you also have the option of using the ready-to-use event specification code snippets in [Console](https://console.snowplowanalytics.com), with or without Snowtype. See below for details.

## Snowtype workflow

The workflow for using Snowtype is:

1. **Define** your events and entities in [Console](https://console.snowplowanalytics.com) or [programmatically](/docs/event-studio/programmatic-management/).
2. **Generate** tracking code by [running Snowtype](/docs/event-studio/implement-tracking/generate-tracking-code/) in your project. It produces typed functions you call instead of constructing event payloads manually.
3. **Track** events using the generated functions in your application code.
4. **Update** when schemas change. Snowtype can [detect new versions](/docs/event-studio/implement-tracking/keep-code-up-to-date/) and regenerate your code.

## Supported trackers

Snowtype generates code for the following Snowplow trackers:

| Tracker                                                 | Language               |
| ------------------------------------------------------- | ---------------------- |
| [Browser](/docs/sources/web-trackers/)                  | TypeScript, JavaScript |
| [JavaScript](/docs/sources/web-trackers/)               | JavaScript             |
| [iOS](/docs/sources/mobile-trackers/)                   | Swift                  |
| [Android](/docs/sources/mobile-trackers/)               | Kotlin                 |
| [React Native](/docs/sources/react-native-tracker/)     | TypeScript             |
| [Flutter](/docs/sources/flutter-tracker/)               | Dart                   |
| [Node.js](/docs/sources/node-js-tracker/)               | TypeScript, JavaScript |
| [Go](/docs/sources/golang-tracker/)                     | Go                     |
| [Java](/docs/sources/java-tracker/)                     | Java                   |
| [Google Tag Manager](/docs/sources/google-tag-manager/) | JavaScript             |

Check out the generated code examples for [data structures](/docs/event-studio/implement-tracking/generate-tracking-code/example-data-structures/) and [event specifications](/docs/event-studio/implement-tracking/generate-tracking-code/example-event-specs/) for each language to see how Snowtype integrates with each tracker.

## Console code snippets

When viewing an [event specification](/docs/event-studio/tracking-plans/event-specifications/) in Console, the **Working with this event** section provides ready-to-use code snippets.

The snippets in the **Implementation** tab show the exact tracking calls needed for each event, including all required properties and entities.

> **Note:** Code snippets are available for the JavaScript tracker only, for event specifications with custom event data structures.

Here's an example snippet for the JavaScript tracker. It provides a `trackSelfDescribingEvent` call for the event specification, with the correct schema references and properties.

The example event specification has an event data structure named `article_click`, and one entity data structure, `article`. The snippet also includes an autogenerated `event_specification` entity. This helps with analysis as it's a direct link between the tracked event and the tracking plan.

To use your snippet, paste it into your application code, and provide the appropriate property values.

```javascript
window.snowplow("trackSelfDescribingEvent", {

    "event": {

        "schema": "iglu:com.example/article_click/jsonschema/1-0-1",

        "data": {

            "name": "", // string - Required - maxLength: 1000

            "location": "", // string - Nullable - maxLength: 1000

        }

    },

    "context": [

        // Entity: article (min: 0)

        {

            "schema": "iglu:com.example/article/jsonschema/3-0-0",

            "data": {

                "publish_date": "", // string - Nullable

                "content_id": "", // string - Nullable

            }

        },

        // System entity. Please do not edit it.

        {

            "schema": "iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-3",

            "data": {

                "id": "0a0ef8bb-314c-4973-8988-f192e8714d68",

                "name": "Article Click",

                "data_product_id": "28a6316a-47fd-473b-b5a1-00c555ba25e4",

                "data_product_name": "Article Performance",

                "data_product_domain": "Marketing"

            }

        }

    ]

});
```

Use the **Show Snowtype code** toggle to display the specific Snowtype function name to call in your tracking implementation. Note that it relies on you having generated the required Snowtype code in your project.

![\&quot;Working with this event\&quot; panel on the Implementation tab showing JavaScript code for a Snowtype-generated trackAddToCartSpec function, with the \&quot;Show Snowtype code\&quot; toggle enabled and a Download button](/assets/images/show-snowtype-code-f0c0e475e699e14327af5e306eee2121.png)

---

# Get started with Snowtype
> Install Snowtype, authenticate with your Snowplow account, and initialize your project.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/install-snowtype/

This page walks you through installing Snowtype and connecting it to your Snowplow account.

To use Snowtype, you'll need:

- [Node.js](https://nodejs.org/en/) version 18 or later
- A Snowplow CDI account
- A Console API key

## Install Snowtype

Navigate to your project directory and install Snowtype as a development dependency:

**npm:**

```bash
npm install --save-dev @snowplow/snowtype@latest
```

**Yarn:**

```bash
yarn add --dev @snowplow/snowtype@latest
```

**pnpm:**

```bash
pnpm add --save-dev @snowplow/snowtype@latest
```

***

This adds a local `snowtype` executable to your project. You can run it with `npx`, `yarn`, or `pnpm`:

```bash
npx snowtype --help
```

> **Note:** The examples on this page use `npx`. Replace with `yarn` or `pnpm` if you prefer.

## Authenticate

Snowtype needs a Console API key to read your event specifications and data structures. You can create or find your API key in [Console](https://console.snowplowanalytics.com/settings) > **Settings** > **Manage organization**.

You need both the **API key** and the **API key ID**.

The recommended approach is to set them as environment variables, either in your shell or in a `.env` file at the root of your project:

**.env file:**

```bash
SNOWPLOW_CONSOLE_API_KEY=your-api-key

SNOWPLOW_CONSOLE_API_KEY_ID=your-api-key-id
```

**Shell variable:**

```bash
export SNOWPLOW_CONSOLE_API_KEY=your-api-key

export SNOWPLOW_CONSOLE_API_KEY_ID=your-api-key-id
```

***

You can also pass the credentials directly as CLI flags (`--apiKey` and `--apiKeyId`), but environment variables are easier to manage and keep credentials out of your command history.

## Initialize your project

Run `snowtype init` to create a configuration file for your project:

```bash
npx snowtype init
```

The command will prompt you for:

- Your **organization ID** from Console
- The **tracker** you want to generate code for, e.g. `@snowplow/browser-tracker`
- The **language** for that tracker, if applicable, e.g. `typescript`
- The **format** (language) you want the configuration file to be in
- The **output path** where Snowtype should write the generated code

To find your organization ID, go to [Console](https://console.snowplowanalytics.com/settings) > **Settings** > **Manage organization**.

Snowtype saves these choices to a `snowtype.config.json` file in your project root. You can edit this file directly later, or pass the values as flags to skip the prompts. See the [configuration reference](/docs/event-studio/implement-tracking/configuration-reference/) for all available options.

---

# Keep Snowtype code up to date
> Update your Snowtype-generated tracking code when schemas and event specifications change, manage your lock file, and automate updates in CI/CD.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/keep-code-up-to-date/

You can check for updates to the schemas and event specifications in your configuration with the `snowtype update` command.

## Keep data structures up to date

When generating code for data structures and Iglu Central schemas, the version number is included in the configuration file. For example, this data structure is version `1-0-0`.

```json
{

  "orgId": "your-org-id",

  "tracker": "@snowplow/browser-tracker",

  "language": "typescript",

  "outpath": "./src/tracking/snowplow",

  "dataProductIds": [],

  "eventSpecificationIds": [],

  "igluCentralSchemas": [],

  "dataStructures": [

    "iglu:com.example/product/jsonschema/1-0-0"

  ]

}
```

Run `snowtype update` to check whether any of your pinned schemas have newer versions available:

```bash
npx snowtype update
```

The command outputs a diff showing the available version updates:

```bash
✔: Valid configuration found

⠧ Checking for updates...✔: Available updates found!

✔: Available Data Structures Updates



 - iglu:com.example/product/jsonschema/1-0-0

 + iglu:com.example/product/jsonschema/2-0-0
```

You can then choose to accept the updates. To skip the confirmation prompt and automatically update your Snowtype configuration file, use the `--yes` flag:

```bash
npx snowtype update --yes
```

Your code won't be regenerated until you run `snowtype generate` again, unless you've configured [`regenerateOnUpdate`](/docs/event-studio/implement-tracking/configuration-reference/#update) in your configuration file.

### Update based on SchemaVer level

For data structure updates, you can filter what `update` shows you based on the [SchemaVer](/docs/fundamentals/schemas/versioning/) bump level. The `--maximumBump` flag sets the highest level of update to include. It defaults to `major`, meaning all updates are shown.

For example, if your configuration pins `iglu:com.acme_company/page_unload/jsonschema/1-0-0` and versions `1-0-1`, `1-1-0`, and `2-0-0` are available:

```bash
npx snowtype update --maximumBump=major

# Shows all updates, including 2-0-0.



npx snowtype update --maximumBump=minor

# Shows 1-1-0 and 1-0-1 only.



npx snowtype update --maximumBump=patch

# Shows 1-0-1 only.
```

You can also set this in your [configuration file](/docs/event-studio/implement-tracking/snowtype-config/) so it applies to every `update` run.

## Keep event specifications up to date

The first time you run `snowtype generate`, Snowtype creates a `.snowtype-lock.json` file next to your configuration file. This lock file pins the exact event specification versions used for code generation.

It has this structure:

```json
{

  "eventSpecifications": {

    "a965caf1-88a6-4a89-9aea-cc92516a9d56": 8,

    "c83f1895-3eb3-469e-a592-a22fabd545b0": 1

  }

}
```

If you've specified tracking plans in your configuration, Snowtype will list event specifications referenced within them individually in the lock file.

> **Note:** Commit `.snowtype-lock.json` to version control. This ensures everyone on your team is generating code from the same versions.

On subsequent runs, `generate` will read from the lock file rather than fetching the latest versions. This ensures reproducible builds, as running `generate` twice produces the same output, even if a schema was updated in between.

If you add new event specifications to your configuration file, and then run `generate` without updating the lock file, Snowtype will skip them:

```bash
Generating...⚠ Warning: Skipping 25 event specification(s) not found in lock file. Run 'update --yes' to add new entries.
```

To check for newer event specification versions, use the `update` command:

```bash
npx snowtype update
```

Use the `--yes` flag to automatically accept all updates and regenerate the lock file:

```bash
✔: Valid configuration found

⠦ Checking for updates...✔: Available updates found!

✔: Available Event Specification Updates



 - 8a4d8cbd-e703-4e06-9484-abc1877771a7: (new)

 + 8a4d8cbd-e703-4e06-9484-abc1877771a7: version 13



ℹ No updates applied

ℹ To apply the available updates automatically run `update` with the --yes flag

ℹ (caution: will overwrite the configuration file)
```

Your code won't be regenerated until you run `snowtype generate` again, unless you've configured [`regenerateOnUpdate`](/docs/event-studio/implement-tracking/configuration-reference/#update) in your configuration file.

### Include draft versions

By default, `update` only checks published event specification versions. If Snowtype isn't detecting your event specification, it might be because it's still in draft mode.

To check for drafts and include the latest draft version, use the `--latestDraft` flag:

```bash
npx snowtype update --latestDraft
```

### Scope update to specific IDs

You can limit the update check to a subset of your configuration:

```bash
# Check specific event specifications only

npx snowtype update --eventSpecs <id1> <id2>



# Check specific tracking plans only

npx snowtype update --dataProducts <id1> <id2>
```

## Clean up stale entries

Over time, your lock file may accumulate entries for schemas you've removed from your configuration. Use `snowtype purge` to clean them up:

```bash
npx snowtype purge
```

This removes any entries in `.snowtype-lock.json` that are no longer referenced by your configuration file.

This is a risky operation. Make sure to commit your lock file beforehand, so you can revert if anything goes wrong.

## Automate with CI/CD

You can run Snowtype in a CI/CD pipeline to catch outdated tracking code before it reaches production. A typical approach:

1. Run `snowtype update --yes` to check for updates, and regenerate the configuration file or lock file if any are found
2. Generate the tracking code with `snowtype generate`
3. If the generated output changes, fail the build or open a pull request with the updated code

The `--yes` flag runs non-interactively, accepting all available updates.

Combine this with [`--disallowDevSchemas`](/docs/event-studio/implement-tracking/generate-tracking-code/#prevent-generation-from-development-schemas) on the `generate` step to also prevent development-only schemas from reaching production:

```bash
npx snowtype update --yes

npx snowtype generate --disallowDevSchemas
```

---

# Configure Snowtype sources and output options
> Configure Snowtype code generation with options for output paths, tracker selection, language settings, and custom templates.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype-config/

Snowtype reads from the configuration file you created with `snowtype init` to determine which schemas to generate code for, and what language the code should be generated in. You can combine multiple source types in a single configuration.

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

## Configuration file structure

The configuration file is a JSON object with the following structure:

```json
{

    "organizationId": "a654321b-c111-33d3-e321-1f123456789g",

    "tracker": "@snowplow/browser-tracker",

    "language": "typescript",

    "outpath": "./src/snowtype",

    "dataProductIds": [

      "a123456b-c222-11d1-e123-1f12345678dp"

    ],

    "eventSpecificationIds": [

      "a123456b-c222-11d1-e123-1f123456789g"

    ],

    "dataStructures": ["iglu:com.myorg/custom_web_page/jsonschema/1-1-0"],

    "igluCentralSchemas": ["iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"],

    "repositories": ["../data-structures"]

}
```

The `organizationId`, `tracker`, `language`, and `outpath` options will be pre-filled based on your answers to the `snowtype init` prompts.

Check out the [configuration reference](/docs/event-studio/implement-tracking/configuration-reference/) page for more details on the available options, including the supported trackers.

## Add sources

Add sources to your configuration manually by editing the file, or with `snowtype patch`. The [command](/docs/event-studio/implement-tracking/command-reference/) will prompt you for the source type and ID, then update your configuration file:

```bash
npx snowtype patch
```

You can also pass in the source details directly:

```bash
# Add event specifications

npx snowtype patch --eventSpecificationIds <id1> <id2>



# Add data products

npx snowtype patch --dataProductIds <id1> <id2>



# Add data structures

npx snowtype patch --dataStructures iglu:com.example/my_entity/jsonschema/1-0-0



# Add Iglu Central schemas

npx snowtype patch --igluCentralSchemas iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0



# Add local schema repositories

npx snowtype patch --repositories ./local-schemas
```

You can disable automatic regeneration in your [configuration file](/docs/event-studio/implement-tracking/snowtype-config/) by setting `regenerateOnPatch` to `false`.

### Tracking plans

A [tracking plan](/docs/event-studio/tracking-plans/) groups related event specifications together. Adding a tracking plan to your configuration generates code for all of its event specifications at once.

To find the tracking plan ID, click **Implement tracking** on the tracking plan page in Console to get the command directly:

![Tracking plan page in Snowplow Console showing the Implement tracking button, which opens a modal with the npx snowtype generate command pre-filled with the tracking plan\&#39;s dataProductIds argument](/assets/images/dp-track-544c276d1535553cf5f930f697af85ad.png)

You can also copy the ID from the URL bar and add it to the `dataProductIds` array in your configuration file. The ID is the last part of the URL after `/data-products/`:

![Browser address bar with the tracking plan UUID highlighted in a red box (52461212-9ea0-4d90-a695-e524c62f7198), with a red arrow pointing to it](/assets/images/dp-id-6c8bd5e400425daa6c524a48efdc129e.png)

```json
{

  "dataProductIds": ["dp-id-1", "dp-id-2"]

}
```

### Event specifications

You can add individual [event specifications](/docs/event-studio/tracking-plans/event-specifications/) if you don't need the full tracking plan. Find the event specification ID on its page in Console. The ID is the last part of the URL after `/event-specifications/`:

![Event specification page in Snowplow Console with the browser URL bar showing the event specification ID as the last segment of the path after /event-specifications/](/assets/images/es-id-0a3367afcd71c89c002445c58795d483.png)

Add the ID to the `eventSpecificationIds` array in your configuration file:

```json
{

  "eventSpecificationIds": ["es-id-1", "es-id-2"]

}
```

### Data structures

To generate code for a specific [data structure](/docs/fundamentals/schemas/), you need its schema tracking URI. Find it on the data structure page in Console, under the **Overview** tab:

![Data structure overview page in Console with the Schema tracking URI field highlighted in a red box, showing iglu.snowplowanalytics.console/data\_structures/jsonschema/1-0-0](/assets/images/ds-url-c93e327627928eba0a66750da09b833a.png)

Add the URI to the `dataStructures` array in your configuration file:

```json
{

  "dataStructures": [

    "iglu:com.example/my_event/jsonschema/1-0-0"

  ]

}
```

### Iglu Central schemas

[Iglu Central](http://iglucentral.com/) hosts schemas that you can use in your tracking. Find the schema tracking URI on the Iglu Central website under **General Information**:

![Iglu Central schema details for button\_click with the Schema tracking URL field highlighted in a red box, showing iglu.snowplowanalytics.snowplow/button\_click/jsonschema/1-0-0](/assets/images/iglu-url-99a7c2aae4d2cd5f818c9dfa1605804a.png)

Add the URI to the `igluCentralSchemas` array in your configuration file:

```json
{

  "igluCentralSchemas": [

    "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"

  ]

}
```

### Local data structure repositories

If you manage schemas locally using [Snowplow CLI](/docs/api-reference/snowplow-cli/data-structures/), you can point Snowtype at your local repository paths. Add them to the `repositories` array:

```json
{

  "repositories": ["./schemas"]

}
```

You can pass in the path to the data structures directly, or the Snowplow CLI directory root.

```json
// This is fine

"repositories": ["./my-snowplow-tracking"]



// This also works

"repositories": ["./my-snowplow-tracking/data-structures"]



// This won't do anything

"repositories": ["./my-snowplow-tracking/data-products"]
```

Provide the path relative to the location of your configuration file.

> **Note:** Snowtype can only create tracking code for local data structures, not for local tracking plans (data products).

---

# Use Snowtype with Google Tag Manager
> Generate tracking code specifically formatted for Google Tag Manager custom JavaScript with Snowtype's GTM target for easier tag implementation.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/working-with-gtm/

Snowtype can generate tracking code specifically formatted for Google Tag Manager (GTM). Instead of importing the code into a JavaScript project, you copy the generated output into a [Custom JavaScript variable](https://support.google.com/tagmanager/answer/7683362?hl=en#custom_javascript) in GTM.

This is useful when your tracking implementation lives entirely within GTM, and you don't have a separate code repository for it.

Benefits of using Snowtype with GTM:

- Generated functions are available on `window.__snowtype` for all tags to use
- Inline JSDoc type definitions for code documentation

## Set up a project

If you already use Snowtype in an existing project, you can switch to GTM output by changing your configuration. Set `tracker` to `google-tag-manager` and `language` to `javascript-gtm`:

```json
{

    "tracker": "google-tag-manager",

    "language": "javascript-gtm"

}
```

Then run `npx snowtype generate` to regenerate.

### Create a new project

If you don't have an existing project, you can create one specifically for GTM. This is a lightweight Node.js project whose only purpose is to run Snowtype and produce the generated output.

You need [Node.js](https://nodejs.org/en/download/package-manager) version 18 or later.

```bash
# Create and enter a project directory

mkdir snowtype-gtm && cd snowtype-gtm



# Initialize the project and install Snowtype

npm init -y

npm install @snowplow/snowtype@latest



# Run the init flow — select 'Google Tag Manager' when prompted

npx snowtype init
```

After completing the `init` flow and adding your desired configuration, generate the code:

```bash
npx snowtype generate
```

Snowtype will create `snowplow.js` and `snowplow.minified.js` files at the path you configured in `outpath`. These files contain the code you'll add to GTM.

## Add the generated code to GTM

The generated `snowplow.js` file is designed to be copied and pasted into a GTM Custom JavaScript variable.

To create the variable:

1. In GTM, go to **Variables** > **New**.
2. Choose **Custom JavaScript** as the variable type.
3. Paste the contents of `snowplow.js` into the code field.
4. Name the variable (for example, `Snowtype Tracking`).

![Creating a Custom JavaScript variable in GTM with Snowtype-generated code](/assets/images/gtm-var-a0b2e5014dc5e14da1288ff2172b2037.gif)

> **Note:** GTM limits Custom JavaScript variables to 20,000 characters. If your generated code exceeds this limit, Snowtype will warn you. Use the `snowplow.minified.js` file instead, which Snowtype generates automatically alongside the full version. If the minified version still exceeds the limit, you can split the code across multiple variables or place it directly in a Custom HTML tag.

## Initialize the tracking code

The Custom JavaScript variable defines a `snowtypeInit()` function but doesn't execute it. To run it, include the variable in a [Custom HTML](https://support.google.com/tagmanager/answer/6107167?hl=en#CustomHTML) tag that fires during page initialization:

```html
<script>

{{Snowtype Tracking}}

</script>
```

Replace `{{Snowtype Tracking}}` with the name you gave to the Custom JavaScript variable.

Once this tag fires, all generated tracking functions become available on the `window.__snowtype` object.

## Call the tracking functions

After initialization, you can call the generated functions from any tag or script on the page. The functions are available on `window.__snowtype`.

For data structures, Snowtype generates `track` and `create` functions:

```js
// Track as a self-describing event

window.__snowtype.trackProductView({ name: "Example", price: 9.99 });



// Create an entity for attaching to other events

var productEntity = window.__snowtype.createProductView({ name: "Example", price: 9.99 });
```

For event specifications, Snowtype generates `Spec` functions that include the event specification metadata automatically:

```js
window.__snowtype.trackAddToCartSpec({

  name: "Example",

  price: 9.99,

  context: [productEntity]

});
```

The bottom of the generated file includes JSDoc type definitions for all functions and their parameters.

---

# Event Studio
> Design and implement behavioral data tracking with schema management, governance, code generation, and tracking plans in Snowplow Console.
> Source: https://docs.snowplow.io/docs/event-studio/

Event Studio is a comprehensive set of tools for designing and implementing behavioral data event tracking. It provides:

- **Schema management**: define and version data structures for events and entities
- **Ownership and governance**: assign ownership and establish data contracts
- **Observability**: monitor data quality and tracking implementation
- **Code generation**: automatically generate tracking code from your designs, using [Snowtype](/docs/event-studio/implement-tracking/)
- **Tracking plans**: document and manage your tracking implementation

The Event Studio UI is included in [Snowplow Console](https://console.snowplowanalytics.com).

These tools help organizations move from ad-hoc tracking implementations to a structured, governed, collaborative approach.

> **Tip:** New to tracking design? Start with our [best practice](/docs/fundamentals/tracking-design-best-practice/) guide to learn how to approach designing your tracking implementation.

## Key concepts

To use Event Studio effectively, you should understand these core concepts:

- **[Events](/docs/fundamentals/events/)**: actions that occur in your systems
- **[Entities](/docs/fundamentals/entities/)**: the objects and context associated with events
- **[Event specifications](/docs/event-studio/tracking-plans/event-specifications/)**: documentation of business events you're tracking
- **[Tracking plans](/docs/event-studio/tracking-plans/)**: logical groupings of related business events with defined ownership

Each tracking plan is associated with one or more [source applications](/docs/event-studio/source-applications/). The events and entities are defined by their [data structures](/docs/event-studio/data-structures/).

This diagram illustrates how these concepts relate to each other within Event Studio:

![Tracking plan overview showing the relationship between tracking plans, event specifications, data structures](/assets/images/tracking-plan-overview-9f48fdeef0c5d4c593b416f9313c174c.png)

This example `Ecommerce Checkout Flow` tracking plan groups two event specifications for ecommerce checkout behavior:

- `Checkout Started` describes a `checkout_started` event, with an associated `cart` entity
- `Product Add To Cart` describes an `add_to_cart` event, with `cart` and `product` entities

The individual event and entity data structures can also be used in other event specifications and tracking plans.

---

# Manage data structures via the API
> Programmatically manage data structures through the API with endpoints for retrieving, validating, and deploying schemas to development and production registries.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/data-structures-api/

The data structures Console API endpoints focus on the main operations in the workflow around:

1. Retrieving existing data structures and their associated schemas
2. Creating or editing new or existing data structures
3. Validating a data structure
4. Deploying a data structure to a registry

## Retrieving data structures

The following `GET` requests allow you to retrieve data structures from both your development and production environment registries.

### Retrieve a list of all data structures

Use this request to:

- Retrieve a list of all data structures
- Retrieve a list of data structures filtered by `vendor` or `name` query parameters

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1`

### Retrieve a specific data structure

Use this request to retrieve a specific data structure by its hash (see 'Generating a data structure hash' below), which is generated on creation.

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1/{dataStructureHash}`

### Retrieve specific version of a specific data structure

Use this request to retrieve all versions of a specific data structure by its hash (see 'Generating a data structure hash' below)

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1/{dataStructureHash}/versions/{versionNumber}`

See the [detailed API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs) for all options.

#### Generating a data structure hash

To use the commands to retrieve information about a specific Data Structure, you need to encode its identifying parameters (`organization ID`, `vendor`, `name` and `format`) and hash it with SHA-256.

**Example:**

| Parameter       | Value                                  |
| --------------- | -------------------------------------- |
| Organization ID | `38e97db9-f3cb-404d-8250-cd227506e544` |
| Vendor          | `com.acme.event`                       |
| Schema name     | `search`                               |
| Format          | `jsonschema`                           |

First concatenate the information with a dash (-) as the separator: `38e97db9-f3cb-404d-8250-cd227506e544-com.acme.event-search-jsonschema`

And then hash them with SHA-256 to receive: `a41ef92847476c1caaf5342c893b51089a596d8ecd28a54d3f22d922422a6700`

## Validation

To validate that your schema is in proper JSON format and complies with warehouse loading requirements, you can use the validation `POST` requests.

`**POST** /api/msc/v1/organizations/{organizationId}/data-structures/v1/validation-requests`

### Example

```bash
curl 'https://console.snowplowanalytics.com/api/msc/v1/organizations/cad39ca5-3e1e-4e88-91af-87d977a4acd8/data-structures/v1/validation-requests' \

  -H 'authorization: Bearer YOUR_TOKEN' \

  -H 'content-type: application/json' \

  --data-binary '{

  "meta": {

    "hidden": false,

    "schemaType": "event",

    "customData": {}

  },

  "data": {

    "description": "Schema for an example event",

    "properties": {

      "example_field_1": {

        "type": "string",

        "description": "the example_field_1 means x",

        "maxLength": 128

      }

    },

    "additionalProperties": false,

    "type": "object",

    "required": [

      "example_field_1"

    ],

    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

    "self": {

      "vendor": "com.acme",

      "name": "example_schema_name",

      "format": "jsonschema",

      "version": "1-0-0"

    }

  }

}'
```

Please note:

- the request's body has two parts:

  - one for data structure metadata as value to the `meta` key
  - one for the schema itself as value to the `data` key

- this example uses the synchronous version of validation that responds with the result immediately. There is also an asynchronous version available that returns a request ID that you can later poll to get the result.

- you can add metadata specific to your organization to the schema as key/value pairs in the `customData` object. See '[Managing Meta Data](#managing-meta-data)' for more information.

## Deployments

The deployment endpoints deal with getting a new or edited version of your data structure into your development and production environments.

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1/{dataStructureHash}/deployments`

`**POST** /api/msc/v1/organizations/{organizationId}/data-structures/v1/deployment-requests`

### Example

```bash
curl 'https://console.snowplowanalytics.com/api/msc/v1/organizations/cad39ca5-3e1e-4e88-91af-87d977a4acd8/data-structures/v1/deployment-requests' \

  -H 'authorization: Bearer MY_TOKEN' \

  -H 'content-type: application/json' \

  --data-binary '{

  "message": "",

  "source": "VALIDATED",

  "target": "DEV",

  "vendor": "com.acme",

  "name": "example_schema_name",

  "format": "jsonschema",

  "version": "1-0-0"

}'
```

Please note:

- This example demonstrates deployment from `VALIDATED` to `DEV`. The method is the same for Production, but you would change the variables where `"source": "DEV"` and `"target": "PROD"`
- The API enforces a workflow of validating, testing on development and then deploying to production. To achieve this you deploy from one environment to another; from (virtual environment) `VALIDATED` to `DEV`, then `DEV` to `PROD`.
- Only users designated as "admin" in the console have the permissions to promote from `DEV` to `PROD`.
- There is a sync option that will return the response of the deployment request directly. Otherwise you can poll for deployment responses using the deployment ID.
- The property for `message` can be sent with a deployment which will capture any change log notes that will be stored against the deployment. (Note that this specific property is a bolt-on feature and might not be available for your account.)

## Managing meta data

Meta data is used to add additional information to a Data Structure.

```json
"meta": {

  "hidden": false,

  "schemaType": "event",

  "customData": {}

}
```

The `hidden` property sets the data structure as visible (true) or not (false) in the console.

The `schemaType` property can be set as null | "event" | "entity".

The `customData` property is mapped as `[string, string]` and can be used to send across any key/value pairs you'd like to associate with the schema.

For example if you wanted to specify departmental ownership through meta data:

```json
"customData": { "department": "marketing" }
```

You can update the meta data for a data structure using the PUT endpoint:

`**PUT** ​/api​/msc​/v1​/organizations​/{organizationId}​/data-structures/v1/{dataStructureHash}​/meta`

## Migrating from the legacy API

The API described above supersedes the legacy one, which was to be found under `/api/schemas`. It offers under-the-hood improvements and a clear path forward to adding further value. To make the transition easier, we have kept the same models for the data being exchanged; it is therefore simply a matter of 1) updating the authentication method, and 2) switching to the new endpoints listed above.

The legacy API is currently tunneled to the new one, so there is no data differences to be expected after the upgrade. We will maintain this facade while supporting customers in upgrading. Your Customer Success Manager and our Engineers will be glad to assist you during this transition.

---

# Manage event specifications via the API
> Programmatically retrieve, create, edit, publish, deprecate, and delete event specifications using the Event Specifications API endpoints.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/event-specifications-api/

The event specifications Console API endpoints allow you to retrieve, create, edit, publish, deprecate, or delete event specifications.

> **Note:** In the previous API version (V1), event specifications were referred to as "tracking scenarios".

## Response Format

The event specifications API follows a specific response format for successful cases (`2xx`) and for scenarios where critical and non-critical errors may occur, such as (`422`) **Unprocessable Entity**.

```json
{

    "data": [

        // Event specifications

    ],

    "includes": [

        // Additional event specifications info

    ],

    "errors": [

        // Warnings or errors

    ]

}
```

- `data`: Contains the event specification/s, depending on the request
- `includes`: Provides additional information, such as the history of event specification changes
- `errors`: Holds a list of errors, which could be of type `Error` or `Warning`. **If the array field contains at least one error of type `Error`, the request will also return a `4xx` status code, indicating that it cannot perform the store operation. Any other severity different from `Error` will result in a `2xx` status code.**

## Compatibility Checks

Certain endpoints conduct a validation to verify the compatibility of a specific event specification event schema, `event.schema`, with the source data structure version referenced by `event.source`. When both `event.schema` and `event.source` are defined in the event specification, the compatibility checks will be executed.

```json
...

"event": {

    "source": "iglu:com.example/ui_actions/jsonschema/1-0-0",

    "schema": {

        "$schema": "http://json-schema.org/draft-04/schema#",

        "description": "Event to capture search submit",

        "type": "object",

        "required": [

            "label",

            "action"

        ],

        "properties": {

            "action": {

                "type": "string",

                "enum": [

                    "click"

                ]

            },

            "label": {

                "type": "string",

                "enum": [

                    "Search"

                ]

            }

        },

        "additionalProperties": false

    }

}

...
```

However, the compatibility check will not only be performed against the version specified by the source data structure (`event.source` field, e.g., `1-0-0`), which we will refer to as the **current** version. It will also be conducted against the latest version available in Iglu, referred to as the **latest** version.

This approach is because it's common for a new event specification to utilize the latest version of the source data structure. However, as this data structure may evolve over time and become incompatible with the `event.schema` defined in the event specification, we provide a method to detect these compatibility issues. Consequently, customers can update the event specification to ensure compatibility.

In cases where an event specification is incompatible or we cannot determine it, some errors will be provided in the `errors` field of the [response](#response-format). These errors alerting of compatibility issues between the event specification and the source data structure will take a similar shape to the one below:

```json
...

"errors": [

    {

        "type":"Warning",

        "code":"SchemaIncompatible",

        "title":"Event specification with id: 59b5e250-91c4-45af-a63d-5f8cd39f4b67, event schema is INCOMPATIBLE with schema with name: test_event, vendor: com.snplow.msc.aws, version: 1-0-13",

        "source":"event.schema"

    }

]

...
```

Compatibility checks can result in three possible values: **Compatible**, **SchemaIncompatible**, or **SchemaUndecidable**.

- If **Compatible**, the event specification is compatible, and no errors will be appended to the `errors` response field
- If **SchemaIncompatible**, the event specification is incompatible against some version. If the check for the **current** version is incompatible, the `type` will be `Error`. For incompatibility with the **latest** version, the `type` will be `Warning`. If the requested operation involves persisting the event specification (create/update), an error of type `Error` will be appended to the response, the status code will be **422 Unprocessable Entity**, and the store operation will not persist. When fetching an event specification, the checks will run for both too, **current** and **latest** versions, and if incompatible, the error type will always be `Warning`, returning status code **200 Ok**.
- If **SchemaUndecidable**, it is indeterminable whether the event specification is compatible with a specific version due to the use of some advanced JSON-Schema features and the high computational cost of checking compatibility. The `type` will always be `Warning`, and the user is responsible for ensuring that the event specification is compatible with the source data structure. A warning will be attached to the `errors` response field.

> **Info:** The algorithm used to perform the compatibility check is based on the [Finding Data Compatibility Bugs with JSON Subschema Checking](https://dl.acm.org/doi/pdf/10.1145/3460319.3464796) paper, authored by Andrew Habib, Avraham Shinnar, and Michael Pradel.

## Retrieve a List of Event Specifications

Use this request to retrieve a list of event specifications within an organization, which will be wrapped into the `data` field of the [response](#response-format).

`GET /api/msc/v1/organizations/{organizationId}/event-specs/v1`

The `organizationId` parameter is required.

### Query Parameters and Filters

You can filter the results based on the following query parameters:

- `dataProductId`: Filters the event specifications associated with a particular tracking plan
- `sourceId`: Filters the event specifications associated with a particular data structure, inferred from the `event.source` field
- `sourceVersion`: Filters the event specifications associated with a specific data structure version when used with `dataStructureId`.
- `withLatestHistory`: When `true`, it will return a list of event specifications, with the latest change per event specification attached to the `includes` array field. The relation between event specifications in `data` and history in `includes` can be determined by `id = eventSpecId`.
- `status`: Filters the event specifications that match the specified status

> **Info:** If no query parameters are provided, it will return all the event specifications for an organization.

## Retrieve a Specific Event Specification

Use this request to retrieve a specific event specification within an organization. The retrieved event specification will be wrapped into the `data` field of the response.

`GET /api/msc/v1/organizations/{organizationId}/event-specs/v1/{eventSpecId}`

> **Info:** This endpoint will trigger [**compatibility checking**](#compatibility-checks) if `event.source` and `event.schema` are defined.

Query parameters `organizationId` and `eventSpecId` are required:

- `withHistory`: When `true`, returns a list with the history for the event specification in the `includes` array field of the response, related to the event specification by its id
- `status`: Filters the event specifications that match the specified status.

## Creating an Event Specification

Use this request to create an event specification within an organization.

`POST /api/msc/v1/organizations/{organizationId}/event-specs/v1`

Query parameter `organizationId` is required.

Request body example:

```json
{

  "spec": {

    "name": "Search",

    "description": "Tracking the use of the search box",

    "event": {

      "source": "iglu:com.example/ui_actions/jsonschema/1-0-0"

    }

  },

  "message": "update"

}
```

The creation form has two fields at the top level, as shown in the example above:

- `message`: An optional field to provide a message
- `spec`: The definition of the event specification, which should comply with the [validations](#validations).

By default, the event specification will be created with `spec.status` set to `draft` and `spec.version` set to `0` if not provided. These values can be changed and managed after creation. Here is an example response:

```json
{

    "data": [

        {

            "id": "5a203ef8-939b-4fd1-914e-f12a3dd1a869",

            "version": 0,

            "status": "draft",

            "name": "Search",

            "description": "Tracking the use of the search box",

            "event": {

                "source": "iglu:com.example/ui_actions/jsonschema/1-0-0"

            }

        }

    ],

    "includes": [

        {

            "author": "39b81015-1bd5-4b37-96c7-3296cabaa36f",

            "message": "initial draft",

            "date": "2023-04-26T14:41:48.708191Z",

            "eventSpecId": "5a203ef8-939b-4fd1-914e-f12a3dd1a869",

            "version": 0,

            "status": "draft",

            "type": "History"

        }

    ],

    "errors": []

}
```

### Validations

- `spec.event.source`: If provided it should match a valid and existing Iglu URI
- `spec.name`: It validates that the `spec.name` of an event specification is unique within the data structure context, inferred from the source data structure `spec.event.source` if provided
- `spec.version`: If provided should be equal or greater than zero
- `spec.status`: If provided should match one of `draft`, `published` or `deprecated`
- `spec.entities`: If provided it will validate that the entities, `spec.entities.tracked` and `spec.entities.enriched`, are not duplicated and that they exist
- `spec.dataProductId`: If provided it will validate that the tracking plans exists. (Coming soon)

> **Info:** This endpoint will trigger [**compatibility checking**](#compatibility-checks) if `event.source` and `event.schema` are defined.

## Editing an Event Specification

Use this request to edit an event specification within an organization. The format of the request and response is the same as during creation.

The `organizationId` and `eventSpecId` parameters are required.

`PUT /api/msc/v1/organizations/{organizationId}/event-specs/v1/{eventSpecId}`

### Publishing an Event Specification

When editing an event specification, it can be published by setting the `status` to `published`. Currently, this will indicate to the event specification consumers (for instance, front-end developers) that the tracking design is ready to be implemented or consumed.

By default, when an event specification is created and no value is provided for `spec.status`, it will be set to `draft`. With this, we suggest an event specification lifecycle that we recommend following, but we allow a certain degree of flexibility to accommodate unique customer use cases. Here is the suggested lifecycle:

```mermaid
graph LR
    style Start fill:#633EB5, stroke:#000000, stroke-width:0px;
    style Draft color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style Published color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style Deprecated color:#89251F, fill:#FDF3F2, stroke:#89251F, stroke-width:1px;
    style Deleted fill:#D63A31, stroke:#000000, stroke-width:1px;

    Start(( )) -->|Create| Draft
    Draft -->|Publish| Published
    Draft -->|Delete| Deleted
    Published -->|Deprecate| Deprecated
    Deprecated -->|Undeprecate| Draft
    Deleted((Deleted))
```

In addition to this lifecycle, and in conjunction with versioning, we enforce that when an event specification is **published**, the versions between two published versions are **discarded**. For example:

Publish new version, before squash:

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style C color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style D color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style E color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    linkStyle default stroke-width:2px,fill:#F2F4F7,stroke:#633EB5,color:#633EB5

    A[Published 1] --> B[Draft 2]
    B[Draft 2] --> C[Draft 3]
    C[Draft 3] --> D[Draft 4]
    D[Draft 4] --> E[Published 5]
```

After discarding intermediate versions:

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    linkStyle default stroke-width:2px,fill:#F2F4F7,stroke:#633EB5,color:#633EB5

    A[Published 1] --> B[Published 5]
```

### Deprecating an Event Specification

When editing an event specification, it can be deprecated by setting the `status` to `deprecated`. This is a way of informing event specifications consumers (for instance, developers) not to rely on the tracking anymore.

### Validations

- `spec.event.source`: If provided, it should match a valid and existing Iglu URI
- `spec.name`: It validates that the `spec.name` of an event specification is unique within the data structure context, inferred from the source data structure `spec.event.source` if provided
- `spec.version`: If provided, it should be equal to or greater than zero, should not exist, and be greater than the last published version
- `spec.status`: If provided, it should match one of `draft`, `published`, or `deprecated`
- `spec.entities`: If provided, it will validate that the entities, `spec.entities.tracked` and `spec.entities.enriched`, are not duplicated and that they exist
- `spec.dataProductId`: If provided, it will validate that the tracking plan exists

> **Info:** This endpoint will trigger [**compatibility checking**](#compatibility-checks) if `event.source` and `event.schema` are defined.

## Deleting an Event Specification

Use this request to delete an event specification within an organization.

`DELETE /api/msc/v1/organizations/{organizationId}/event-specs/v1/{eventSpecId}`

> **Warning:** Please note that this action is irreversible and will permanently delete the event specification.

---

# Programmatic management using CLI or API
> Automate Event Studio workflows using Snowplow CLI for git-ops integration or the APIs for custom tooling and CI/CD pipelines.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/

Data structures and tracking plans can be managed programmatically, enabling git-ops workflows, CI/CD integration, and custom tooling.

Partnered with other tools like the data structures [CI tool](/docs/testing/data-structures-ci-tool/) and the testing pipeline [Snowplow Micro](/docs/testing/snowplow-micro/), it's possible to have a very robust and automated data structure workflow that ensures data quality upstream of data hitting your pipeline.

## Snowplow CLI

The [Snowplow CLI](/docs/api-reference/snowplow-cli/) provides file-based workflows for git-ops integration. It has commands to:

- [Send events](/docs/api-reference/snowplow-cli/send-events/) to a Collector from the command line, for testing or scripting
- Download and upload [data structures](/docs/api-reference/snowplow-cli/data-structures/) as local YAML/JSON files
- Manage [tracking plans](/docs/api-reference/snowplow-cli/tracking-plans/), event specifications, and source applications
- Validate resources before publishing
- Integrate with version control and code review workflows

The [Snowplow CLI MCP server](/docs/llms-support/cli-mcp-server/) enables AI assistants to interact with your tracking plan resources through natural language.

## Console API

Use the Console REST API for direct programmatic access to tracking plans and data structures. This is useful for integration with other tools, or when you need more control than file-based workflows allow.

It has endpoints for three resource types:

- [Data structures](/docs/event-studio/programmatic-management/data-structures-api/) (`/data-structures/v1`): retrieve, validate, and deploy schemas to development and production registries
- [Tracking plans](/docs/event-studio/programmatic-management/tracking-plans-api/) (`/data-products/v2`): create and update tracking plans, view change history, and manage subscriptions
- [Event specifications](/docs/event-studio/programmatic-management/event-specifications-api/) (`/event-specs/v1`): create, publish, deprecate, and delete event specifications within tracking plans

You can explore all available endpoints in the [Swagger API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs).

> **Note:** By default, Snowplow pipelines use Iglu Server schema registries. Each pipeline has a development and a production Iglu Server instance.
>
> The Console API only works with these registries. If you're using a custom static S3 registry instead, you'll need to update your registry manually.

### Authentication

Each request requires your organization ID and an authorization token. You can find your organization ID [on the **Manage organization** page](https://console.snowplowanalytics.com/settings) in Console.

Follow the instructions in the [Account management](/docs/account-management/) section to obtain an access token for API authentication.

To be able to post sample requests in the [Swagger API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs), click the `Authorize` button at the top of the document and authorize with your token. The value for the token field in each individual requests is overwritten by this authorization.

---

# Managing tracking plans via the API
> Programmatically manage tracking plans through the API with endpoints for creating, updating, retrieving, and managing subscriptions for automated workflows and version control integration.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/tracking-plans-api/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

Use the tracking plans Console API endpoints to programmatically manage your tracking plans.

## Retrieving Information about Tracking Plans

The following `GET` requests are designed to allow you to access information about tracking plans.

### Retrieve a List of All Tracking Plans

To retrieve a comprehensive list of all tracking plans in your organization, you can use the following GET request:

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2`

Path parameter `organizationId` is required.

### Retrieving Information about a Specific Tracking Plan

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}`

Path parameters `organizationId` and `dataProductId` are required.

When retrieving a tracking plan, it could also contain an array field `data[].tracking_scenarios` that will include the `id` and `url` of the associated event specifications. For example:

```json
"data": [

  ...

  "eventSpecs": [

    {

      "id": "d1336abc-1b60-46f7-be2d-2105f2daf283",

      "url": "https://console.snowplowanalytics.com/api/msc/v1/organizations/f51dada7-4f11-4b6a-bbbd-2cf6a3673035/event-specs/v1/d1336abc-1b60-46f7-be2d-2105f2daf283"

      }

  ]

  ...

]
```

Under the json path `includes.tracking_scenarios`, the API will also attach associated event specifications in their entirety:

```json
"includes": {

  ...

  "eventSpecs": [

    "id": "d1336abc-1b60-46f7-be2d-2105f2daf283",

     ...

  ]

  ...

}
```

### Retrieve History Information for a Tracking Plan

If you wish to retrieve the change log of a specific tracking plan, you can use the following GET request:

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}/history`

You can pass several parameters to control the result of the response:

- **before**: returns records equal or less than the timestamp in the ISO-8601 format
- **limit**: limits the number of records
- **offset**: skip the first N results
- **order**: order of returned records, `asc` or `desc`. Defaults to `desc`.

Path parameter `organizationId` is required.

## Creating and updating Tracking Plans

### Creating a Tracking Plan

This `POST` request allows you to create a new tracking plan within an organization.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2`

The request body is mandatory and should be in JSON format. The minimum payload would be a JSON with only the `name` of the tracking plan. The remaining fields are optional and not required on creation. Example:

```json
{

  "name": "Performance tracking",

  "description": "Tracks performance",

  "domain": "Marketing",

  "owner": "IT department",

  "accessInstructions": "The data can be accessed in the warehouse, in the atomic.events table"

}
```

> **Note:** The name of your tracking plan must be unique to ensure proper identification and avoid conflicts.

### Updating a Tracking Plan

Use this request to update a tracking plan. The `dataProductId` is required, along with a valid request body.

The minimum payload on update would be the same as on creation but with the addition of the required `status` field. On creation, by default, it will set the `status` to `draft`.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}`

> **Note:** The name of your tracking plan must be unique to ensure proper identification and avoid conflicts.

See the [detailed API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs) for all options.

### Delete a Tracking Plan

Use this request to delete a tracking plan. The `dataProductId` and `organizationId` are both required.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}`

## Subscription Management for Tracking Plans

### Retrieve All Subscriptions for a Tracking Plan

To retrieve all subscriptions for a tracking plan, use the following request. The `organizationId` and `dataProductId` are required.

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions`

### Add a Subscription

To add a subscription for a tracking plan, use the following request. The `organizationId`, `dataProductId` and a valid request body are required.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions`

The following is the minimum accepted payload. It will create a subscription for the user who issues the request, as inferred by the JWT in the request headers.

```json
{

  "reason": "Get notified on breaking changes",

  "receiveNotifications": true

}
```

If you want to subscribe a different user you will need to populate an additional field, `recipient`, with that user's email address.

When a subscription is created, it will send a confirmation email to the recipient (default user or third user). Clicking the confirmation link in that email will direct the recipient to the following URL and mark the subscription as confirmed:

`**POST** /organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}/actions/confirm`

Once a subscription is created and the email has been confirmed, the subscriber will start receiving a daily email digest referencing all the tracking plans that had changes in the last 24 hours.

### Update a Subscription

To update a subscription for a specific tracking plan, use the following request. Path parameters `organizationId`, `subscriptionId`, `dataProductId`, and a valid request body are required.

`**PUT** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}`

### Delete a Subscription

To delete a subscription for a specific tracking plan (unsubscribe action), use the following request. Path parameters `organizationId`, `subscriptionId`, `dataProductId`, and a valid request body are required.

`**DELETE** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}`

### Resend a Subscription Confirmation Email

To resend a subscription confirmation email, use the following request. Path parameters `organizationId`, `subscriptionId`, `dataProductId` are required.

`**POST** ​/api​/msc​/v1/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}/actions/resend-confirmation`

### Integration with the SDK Generator

To send emails with instructions for the SDK generator, use the following request. Path parameters `organizationId` and `dataProductId` and a valid request body are required.

`**POST** /organizations/{organizationId}/data-products/v2/{dataProductId}/share-instructions`

---

# Query your data
> Use example SQL queries from Console to retrieve and analyze your event data in your data warehouse.
> Source: https://docs.snowplow.io/docs/event-studio/query-sql/

When viewing an [event specification](/docs/event-studio/tracking-plans/event-specifications/) in Console, the **Working with this event** section provides ready-to-use code snippets.

The **Querying** tab provides example queries to help you retrieve and analyze your event data. Choose your warehouse to see appropriately optimized SQL.

![\&quot;Working with this event\&quot; panel on the Querying tab showing a Snowflake SQL query that selects flattened event and context columns from atomic events, filtered by event name and vendor](/assets/images/sql-example-d64be1abac9a42e1162f0ed311e2db80.png)

---

# Organize data sources with source applications
> Document and manage your tracking implementation across different applications with source applications in Event Studio.
> Source: https://docs.snowplow.io/docs/event-studio/source-applications/

For data collection, you will often have different sources of information that correspond to applications designed for a particular purpose. These are what we refer to as source applications.

> **Tip:** A guideline, which will address your needs most of the time, is to think of a source application as an independently deployable application system. For example:
>
> - An Android mobile application
> - An iOS mobile application
> - A web application
>
> This will let you best manage changes you make to the available application entities and make sure it reflects as closely as possible the current data state.

To illustrate, consider Snowplow. We can identify several applications designed for distinct purposes, each serving as a separate data source for behavioral data, or in other words, a source application:

- The Snowplow website that corresponds to the application served under `www.snowplow.io`
- The Console application that is served under `console.snowplowanalytics.com`
- The documentation website serving as our information hub for all things related to our product, served under `docs.snowplow.io`

Source applications are a foundational component that enables you to establish the overarching relationships that connect application IDs and [application entities](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) and [tracking plans](/docs/event-studio/tracking-plans/).

## Application IDs

Each source application should have a unique application ID to distinguish it in analysis. For each of these applications you would set up a unique application ID using the [`app_id`](/docs/events/ootb-data/app-information/#application-atomic-event-properties) field to distinguish them later on in analysis.

> **Tip:** We often see, and recommend as a best practice, setting up a unique application ID for each deployment environment you are using. For example, `${appId}-qa` for staging and `${appId}-dev` for development environments.

## Application entities

Application entities, also referred to as [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/), are a set of entities that can be sent with every event recorded in the application. Using source applications you can document which application entities are expected. This is useful for tracking implementation, data discovery, and preventing information duplication in tracking plans.

> **Info:** Since application entities can also be set conditionally, you can mark any of them as optional with a note to better understand the condition or any extra information required. The method for conditionally adding an application entity is through [rulesets](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#rulesets), [filter functions](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#filter-functions) and [context generators](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#context-generators).

## Initialize tracking with the tracker configuration

Once you have created a source application, you can use the **Set up tracking** tab to configure tracking and generate ready-to-use code snippets. This guided configuration simplifies the instrumentation process and helps you start receiving events.

> **Info:** The Set up tracking tab currently supports the JavaScript tracker. You can always customize your tracking further by referring to the [tracker documentation](/docs/sources/).

### Configure tracker settings

The Set up tracking tab provides a visual interface to configure your tracker and generates code snippets based on your selections.

#### Initialize tracker

Configure the basic tracker settings:

- **Collector URL**: the endpoint where your events will be sent
- **App ID**: select one of the application IDs associated with your source application

![A screenshot from Console showing the initialize tracker section, with dropdowns for Collector URL and App ID](/assets/images/initialize-tracker-96764a0c520edf581afece137c0282dd.png)

#### Automatic tracking

Enable out-of-the-box tracking features to capture common user interactions without additional code:

- **Page views**: automatically track when pages are viewed
- **Link clicks**: capture clicks on links
- **Form interactions**: track form submissions and field interactions
- **Page pings**: monitor user engagement with periodic activity pings

Toggle these features based on your tracking requirements.

![A screenshot from Console showing automatic tracking configuration](/assets/images/automatic-tracking-8a40a05374e35316c3710f66a6ca9759.png)

#### Implementation

The code snippet at the bottom of the Set up tracking tab updates in real-time as you modify settings. Copy the code snippet and integrate it into your application to begin tracking.

Choose your implementation method:

**JavaScript (tag):**

Add a `<script>` tag to your website.

The SDK URL is pre-filled with the latest version from a CDN but can be edited if you are hosting on your own domain.

![JavaScript tag implementation showing an editable SDK URL field and a generated script tag code snippet](/assets/images/tag-implementation-f6870244702c1b66b03cedc8d63f4d06.png)

**Browser (npm):**

Install the tracker as an npm package for bundled applications and any required plugins using your preferred package manager.

![Browser npm implementation showing package installation commands and import code snippet](/assets/images/npm-implementation-a96a56c0913305c882771df8a58b5f8b.png)

***

---

# Managing tracking plans using Console
> Create and manage tracking plans in Snowplow Console with templates, event specifications, source applications, and automatic code generation for tracking implementation.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/create-and-manage/

To create a new tracking plan, navigate to the "Tracking plans" section from the navigation bar and click the "Create tracking plan" button.

![Tracking plans list page showing six tracking plans with their domain, status, event volume, event spec count, and last modified date, with a \&quot;+ Create tracking plan\&quot; button in the top right](/assets/images/create-tracking-plan-v2-7dd09e1670e4fadd80c1d27912b423a4.png)

A modal will appear on the page, giving you the possibility to quickly create a tracking plan by using one of the existing templates or create one from scratch.

![\&quot;Create tracking plan\&quot; dialog offering a \&quot;Create new\&quot; option and three templates: E-commerce Web (11 events), Base Web (5 events), and Media Web (24 events)](/assets/images/create-tracking-plan-modal-effaa5d8fbffc01a04a09662fc6a9a46.png)

After selecting "Create new" a form will appear on the page. Enter your tracking plan information and click "Create and continue" to navigate to the event specification page.

> **Note:** The name of your tracking plan must be unique to ensure proper identification and avoid conflicts.

The next page allows you to create multiple event specifications. You can click on any row to enter the details on this screen, or you can complete the information later.

When clicking on an event specification row, a page will allow you to enter additional information into separate modals:

- **Event information**; describes information such as which applications the event fires in
- **Event data structure**; determines how your data is structured and define the types of properties of the event
- **Entity data structures**; describes which entities to attach to the event when it is triggered
- **Event triggers**; specify the places and circumstances under which the event is triggered
- **Properties**; specify instructions about how each schema property should be set for this event specification

The breadcrumb navigation allows you to quickly navigate to the tracking plan overview as well as to the list of tracking plans. Alternatively, you can access the list of available tracking plans by clicking `Tracking plans` prominently displayed in the navigation bar on the left.

In the image below, you can see an example of a tracking plan. It not only provides an overview of all the event specifications but also allows you to access three important pieces of functionality.

- **Share**; allow other members of your organization to access the tracking plan
- **Subscribe**; receive notifications of any changes in the tracking plan
- **Implement tracking**; automatically generate the code for your tracking plan to be included in your application (to learn more visit [Code Generation - automatically generate code for Snowplow tracking SDKs](/docs/event-studio/implement-tracking/))

> **Note:** Sharing and subscribing is only available for users registered in Snowplow Console.

![E-commerce Web tracking plan overview showing general information, E-commerce domain ownership, and an event specifications table with four draft events (Add to cart, Checkout step, Internal promotion click, Internal promotion view) all using the snowplow\_ecommerce\_action 1-0-2 data structure](/assets/images/tracking-plan-overview-0d2201d440d341fe7fff177dd83fef8d.png)

![Add to cart event specification page showing event description, four tracked application IDs, the snowplow\_ecommerce\_action 1-0-2 data structure with a \&quot;type\&quot; property set to \&quot;add\_to\_cart\&quot;, and product and cart entity data structures](/assets/images/event-specification-details-1c54133bbc47f99ca82386d40f3f6284.png)

If you need to edit a tracking plan at any time, select it from the tracking plans listing accessible from the main menu.

## Upgrading Event Specification Instructions

When working with Event Specifications of a Tracking Plan, it's essential to account for the evolution of underlying [Data Structures](/docs/event-studio/data-structures/). Data Structures define reusable JSON schemas, which can be referenced by different Event Specifications (events and entities). Each Event Specification may contain instructions, which rely on a specific version of a Data Structure, adding another layer to specialize or constraint Event Specifications in a more granular way.

### Versioning of Data Structures

As data and events evolve, Data Structures may be updated to new versions, which can be either compatible or incompatible with previous ones. These changes may cause potential conflicts with the instructions in Event Specifications (both events and entities) that reference an older version of the Data Structure.

### Semi-Automatic Upgrade of Event Specifications via the UI

To streamline the process of upgrading an Event Specification to the latest version of a Data Structure, we've implemented a mechanism that allows you to update Event Specification instructions through the UI. Here's how it works:

When a new version of a Data Structure becomes available, the system will indicate that the event or entities referenced by the data structure has a new version available, showing an **'Upgrade'** button in the UI.

![Event data structure section with a warning that a new version of the data structure is available, showing add\_to\_cart at version 1-0-0 and an Upgrade button](/assets/images/upgrade-event-specification-warning-c04429de0f17e3b4ddd05404d6d3b691.png)

Clicking the button navigates to a new page, informing the user of the new version they are upgrading to, along with a **'View Changes'** button.

![\&quot;Upgrade Add Product\&quot; page warning that add\_product\_to\_cart will be upgraded from version 1-0-0 to 1-0-1, with a \&quot;View changes\&quot; button and Upgrade and Cancel actions](/assets/images/upgrade-event-specification-page-194f756d9e7d4eef25df6d940b4949bc.png)

When clicked it will show the differences between the current version of the Data Structure and the one the user intends to upgrade to.

![Side-by-side diff view showing JSON schema changes between the current version 1-0-0 and the latest version 1-0-1 of a data structure, with additions highlighted](/assets/images/upgrade-event-specification-diff-7cb68782706e2707a441dbb890508b3a.png)

At the bottom, a button will allow users to confirm the upgrade. One of two things can happen when the upgrade is confirmed:

#### 1. Successful automatic upgrade

- If the Event Specification instructions are compatible with the new Data Structure version, the system will automatically upgrade the Event Specification to the latest version of the Data Structure.
- All instructions will be updated seamlessly without any further user intervention.

![\&quot;Upgrade Add Product\&quot; page showing a green \&quot;Data structure updated\&quot; success message with a \&quot;Return to Event Specification\&quot; button](/assets/images/success_upgrade-ac3deb29a3bcf0154246aaf4d2416f03.png)

#### 2. Conflict detection and resolution

If the new version of the Data Structure introduces incompatibilities with the existing Event Specification instructions, the system will flag the conflicting properties.

- The UI will prompt the user to resolve these conflicts before the Event Specification can be upgraded.

- The conflict resolution UI provides options to the user to modify or delete each instruction depending on the type of incompatibility:

  - **Remove conflicting instructions**: If a specific property is no longer present in the new Data Structure.
  - **Modify conflicting instructions**: If a property in the new Data Structure has been changed in an incompatible way (e.g., type change, added/removed enum values, added pattern, etc.).

![\&quot;Upgrade Add Product\&quot; page showing a conflict where the upgrade from 1-0-1 to 2-0-0 couldn\&#39;t complete because the Quantity property instruction is incompatible with the new version and requires manual editing](/assets/images/conflict_resolution-1eaa5c52b9eb95d0feb3f26d2d42e5f0.png)

This mechanism ensures that teams can benefit from updated Data Structures while maintaining the integrity and accuracy of their Event Specifications. Users are empowered to make informed decisions during the upgrade process, with clear visual cues and options to handle conflicts effectively.

---

# Event specification inference
> Event specification inference enables the Snowplow pipeline to automatically match incoming events to your published event specifications, surfacing business context and volume metrics without changes to your tracking implementation.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/event-specification-inference/

Event specification inference allows the Snowplow pipeline to automatically match incoming events against your published [event specifications](/docs/event-studio/tracking-plans/event-specifications/). Once a specification is published, the pipeline can attach business context to matching events and surface volume and recency data in the Console — without any changes to your tracking implementation.

## Understand event specification statuses

Each event specification uses an explicit publishing model, replacing the previous "Live" status that the Console assigned automatically when it first observed matching events. Each specification has one of three statuses:

- **Draft**: the specification is being edited and is not yet active in the pipeline. The pipeline does not match events against it, and no inference occurs.
- **Publishing**: a transitional state, lasting a few minutes, while the pipeline propagates the specification. You do not need to take any action during this phase.
- **Published**: the specification is active. The pipeline matches incoming events against it, attaches an `event_specification` entity to it, and surfaces volume data and "last seen" timestamps in the Console.

The tracking plan list view reflects the status of the specifications it contains. A tracking plan shows **Published** if all of its specifications are published, and **With Drafts** if any specification remains in draft.

## Publish an event specification

Publishing makes a specification active in the pipeline and enables inference. You can trigger publishing from two places:

- From the tracking plan view: click the **Publish All** button above the list of specifications. This publishes all specifications in the plan that are currently in Draft status.
- From within an individual event specification: click the **Publish** button at the top of the specification page.

The publishing flow also checks that any [data structures](/docs/fundamentals/schemas/) your specification references are available in the production pipeline before activating it.

### Synchronize data structures during publishing

Before publishing completes, the Console checks whether all [data structures](/docs/fundamentals/schemas/) referenced by the specification — either the event data structure or any entity data structures — have been promoted to production. If any have not, the publishing flow identifies them and requires you to promote them in the same step. This ensures the pipeline can perform inference correctly as soon as the specification reaches Published status.

## Understand how inference works

Once a specification is published, the pipeline evaluates every eligible incoming event against it. Events sent using Snowtype's [automatic tracking features](/docs/event-studio/implement-tracking/) already contain an `event_specification` entity, so the pipeline bypasses inference for them entirely. All other events are evaluated using the following criteria, in order:

1. **Event schema**: the event's schema must match the event data structure defined in the specification.
2. **Entity set**: the event must carry all [entities](/docs/fundamentals/entities/) listed in the specification according to the cardinality rules. Extra entities with different schemas on the event are ignored.
3. **Property instructions**: any property-level instructions defined on those data structures in the specification — for example, `category = "product"` — must be satisfied.

The pipeline does not match on `appId`, environment, or any other source attribute. A single incoming event can match more than one specification if multiple published specifications share overlapping definitions.

When a match occurs, the pipeline:

- Attaches an [entity](/docs/fundamentals/entities/) to the event containing the specification name and ID, making the business context available for downstream consumers.
- Records the event against the specification in the Console, updating the total volume count and "last seen" timestamp.

### Example

Consider a specification named "Product page view" that defines:

- **Event**: `page_view` (`iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0`)
- **Entity**: `product` (`iglu:com.acme/product/jsonschema/1-0-0`) with a rule `category = "electronics"`

An incoming `page_view` event carrying a `product` entity where `category = "electronics"` matches the specification. The pipeline attaches an `event_specification` entity to the event and increments the specification's volume count in the Console.

The same `page_view` event carrying a `product` entity where `category = "clothing"` does not match, because the entity rule is not satisfied. A `page_view` event with no `product` entity also does not match, because the required entity is absent.

Inference is the path the pipeline takes for events that do not arrive with an `event_specification` entity already attached. For events that do, the pipeline runs [event specification validation](/docs/event-studio/tracking-plans/event-specification-validation/) against that specification instead, which produces explicit per-event findings when an event fails to conform to its rules.

> **Tip:** You do not need to change your tracking implementation to benefit from inference. Events already flowing through your pipeline will be matched against newly published specifications automatically.

---

# Event specification validation
> Event specification validation checks events that arrive with a specification entity attached against the rules defined in your published specifications. When an event fails validation, the pipeline attaches an entity describing the failure.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/event-specification-validation/

Event specification validation checks whether incoming events conform to the rules defined in the [event specifications](/docs/event-studio/tracking-plans/event-specifications/) you have published.

The pipeline runs validation on events that arrive with an `event_specification` entity already attached, typically from a tracker using [Snowtype](/docs/event-studio/implement-tracking/) version 0.17.0 or later (see [Validate against a new specification version](#validate-against-a-new-specification-version) for the upgrade steps). For events that arrive without one, the pipeline runs [event specification inference](/docs/event-studio/tracking-plans/event-specification-inference/) instead. Each event takes one path or the other; the two paths are mutually exclusive per event. Inference never produces a validation entity, even when an event would have failed validation.

When an event fails validation, the pipeline attaches an [entity](/docs/fundamentals/entities/) to it describing the failure. The pipeline still enriches and delivers events that fail validation alongside successful ones; only the validation entity reflects the failure. You don't need to change your tracking implementation, as validation runs against newly published specifications immediately.

> **Note:** The pipeline still delivers events that fail validation as enriched events. The pipeline records the failure in the validation entity, but does not route the event to [failed events](/docs/fundamentals/failed-events/).

In your warehouse, three cases are possible:

- **Failed validation**: the event has both an `event_specification` entity and an `event_specification_validation` entity. Inspect the validation entity for the specific findings.
- **Passed validation, or matched by inference**: the event has an `event_specification` entity but no `event_specification_validation` entity. Either the event was Snowtype-tracked and passed validation, or the pipeline matched it to a specification by inference.
- **Not associated with a specification**: the event has neither entity. No specification was attached at tracking time, and the pipeline did not match the event to a published specification.

## Validation entity

The pipeline attaches an `event_specification_validation` entity to events that fail validation.

### `event_specification_validation`

**Type:** Entity

Validation result for an event against its declared event specification version.

**Properties:**

| Property            | Description                                                                      |
| ------------------- | -------------------------------------------------------------------------------- |
| `isValid` _boolean_ | _Required._ Whether the event passed validation against its event specification. |
| `errors` _array_    | _Optional._ A list of validation errors.                                         |

The pipeline only attaches the entity on failure, so `isValid` is always `false`.

Each finding in the `errors` array includes a human-readable `message`, the Iglu `schema` URI of the entity or event that caused the error (when applicable), and the JSON `path` to the property that failed validation (when applicable).

Events that pass validation receive no entity. Use the entity's presence in your warehouse to identify events that failed validation.

## Understand what the pipeline validates

The pipeline evaluates each event against three categories of rule defined in its specification, in addition to the [schema validation](/docs/fundamentals/schemas/) that always runs as part of enrichment:

1. **Event property instructions**: property-level instructions defined on the event data structure, such as expected values or ranges
2. **Entity cardinality**: how many of each entity listed in the specification must be present on the event. Cardinality is checked per entity schema.
3. **Entity property instructions**: property-level instructions defined on each entity data structure listed in the specification

If the event payload itself fails its schema validation, the pipeline doesn't evaluate event property instructions because the payload is no longer available to the validation step. Entity cardinality and entity property instructions still run, and the pipeline attaches a validation entity only if one of those entity checks fails.

### Examples

Consider a published specification "Add to cart" that defines:

- Event: `add_to_cart` (`iglu:com.acme/add_to_cart/jsonschema/1-0-0`) with rule `currency = "USD"`
- Required entity: `product` (`iglu:com.acme/product/jsonschema/1-0-0`) with cardinality of exactly one

The `add_to_cart` data structure's schema permits `currency` to be either `"USD"` or `"EUR"`. The specification narrows it to `"USD"`, so the event property instruction catches values that the underlying schema would otherwise accept.

#### Example 1

An `add_to_cart` event arrives with:

- An `event_specification` entity attached referring to this specification
- An event payload where `currency = "USD"`
- A single `product` entity

The event passes all checks: no validation entity is attached.

#### Example 2

An `add_to_cart` event arrives with an event payload where `currency = "EUR"`.

The data structure's schema accepts the value, but the specification's event property instruction does not. The pipeline attaches a validation entity with `isValid: false`, and an error identifying the event schema and the violated instruction.

#### Example 3

An `add_to_cart` event arrives with:

- An event payload where `currency = "USD"`
- Two `product` entities

The event fails the entity cardinality check. The pipeline attaches a validation entity with `isValid: false`, and an error identifying the `product` entity schema and the violated cardinality rule.

## Validate against a new specification version

A Snowtype-tracked event declares a specific `(id, version)` pair in its `event_specification` entity, and the pipeline validates each event against that event specification version. Snowtype attaches the `version` field starting in version 0.17.0; events tracked with earlier Snowtype versions don't produce a validation entity.

> **Info:**&#x31;) Update the Snowtype dependency in your package manager or CI configuration to 0.17.0 or later
> 2) Regenerate the tracker code with `snowtype generate`, either manually or as part of your build pipeline
> 3) [Publish](/docs/event-studio/tracking-plans/event-specifications/) the relevant event specifications; validation doesn't run against drafts
> 4) Deploy the regenerated tracker code

When you publish a new version of a specification, events from existing tracker code continue to declare the previous version, validating against its instructions. The new version applies once you regenerate your tracker code with [Snowtype](/docs/event-studio/implement-tracking/) and deploy the updated tracker code, after which new events declare and validate against the new version.

Different versions of a specification can coexist. Each event validates against the version it declares, whether the variation comes from a rolling update, from different applications using different versions, or from different deployment stages.

---

# Manage event specifications using Console
> Create and manage event specifications in Snowplow Console UI as documented events within tracking plans with data structures and implementation details.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/event-specifications/

Event specifications are complete definitions of business events. They define what data should be tracked, when the event occurs, along with additional metadata about the event such as ownership.

They act as data contracts between teams. When you create an event specification, you are defining exactly what data your applications should send, what your data warehouse will receive, and what your downstream consumers can rely on.

> **Note:** You can only make event specifications within tracking plans. Tracking plans are wrappers for related event specifications. Each event specification belongs to a single tracking plan.

Once an event specification is ready, you can publish it to make it active in the pipeline. Publishing enables two features: [event specification inference](/docs/event-studio/tracking-plans/event-specification-inference/) and [event specification validation](/docs/event-studio/tracking-plans/event-specification-validation/). Inference automatically matches incoming events to your published specifications, surfacing volume and observability data in the Console. Validation applies to events that arrive with a specification entity attached: if an event doesn't conform to the specification's instructions, a validation entity is attached to flag the issue.

Each event specification has an event [data structure](/docs/event-studio/data-structures/), plus optional entity data structures. You can add instructions for how to populate each property, when to trigger the event, and how many of each entity to expect. You can also select which version of each data structure to use.

## Create an event specification in Console

To create a new event specification, follow these steps:

1. Select a tracking plan
2. Click the **Create event** button
3. A dialog will appear, prompting you to enter a name for your event specification and click **Save and continue**
4. Your first event specification will be displayed on the page

![E-Commerce Web User Interactions Data tracking plan overview with an arrow highlighting the Create event button at the bottom of the event specifications list](/assets/images/create-event-specification-1fb10126179bd86b5036a979b2d44b39.png)

To add more information or modify an existing event specification, follow these steps:

1. Navigate to the appropriate tracking plan
2. Select the desired event specification
3. This action will open an overview of the selected event specification containing the details that have been added to date

This interface is divided into focused sections; explore each section below for more details.

![Product Added to Cart event specification showing event description, source application, the cart\_action 1-0-0 data structure with a type property required to be \&quot;add\&quot;, user and product entity data structures, and an Add to Cart button trigger](/assets/images/event-specification-overview-f3c4beeb305a7329dbc7ed8376a57f0b.png)

### Event information

This section provides essential meta-information for your event specification, including the event name, description, and the applications in which this event is tracked.

### Event data structure

This section defines the event data structure that this event will validate against as it is processed by your pipeline.

You can choose from two types of data structures:

- **Standard**: provided by the Snowplow tracker by default
- **Custom**: provided by your organization

![\&quot;Event data structure\&quot; selection dialog showing the Standard tab with four options: link\_click (1-0-1), page\_ping (1-0-0), page\_view (1-0-0), and screen\_view (1-0-0), each with a \&quot;View properties\&quot; link](/assets/images/event-data-structure-00f3d61c035b8d6e2261531677f57083.png)

### Entity data structures

Within this section, you have the flexibility to define the entities that should be associated with the event when it is triggered.

To facilitate making an informed selection of your entities, you can view a detailed breakdown of the properties associated with the chosen entities (and its version) by a specific version.

![\&quot;Entity data structures\&quot; selection dialog showing the Custom tab with six entities listed, the product entity selected, and a \&quot;Select entities\&quot; button](/assets/images/entity-data-structures-a2728bea5606b9e3aa9eb473ab676c56.png)

### Event triggers

This section defines the locations and circumstances under which this event is triggered.

To create your first trigger, simply select the **Add trigger** button. You can edit or delete an existing trigger by clicking the dedicated buttons beside each entry in the triggers list.

A dialog will appear, allowing you to upload an image and provide additional context, such as the URL to the page on which this trigger applies.

![\&quot;Event trigger\&quot; dialog with fields for uploading a screenshot, adding trigger notes, specifying which applications the trigger applies to, and providing a URL for the page where the trigger fires](/assets/images/event-triggers-f58df8f12d6e3816cbe742f2b7b523c8.png)

### Properties

This section allows you to specify how each property for a selected event or entity data structure should be populated.

The dialog displays the list of properties for the selected data structure will be listed here, with the ability to provide the exact value/s or a description of how to populate these properties when the event is triggered.

You can configure detailed instructions for any of the properties shown in the list by clicking the **Add instruction** or **Edit** buttons. Once you have selected the type of instruction you wish to add/edit and have filled in the required input fields, you are then able to click **Save and update instruction** and return to the properties list.

> **Note:** Instructions for required properties are added by default and can be edited but they cannot be deleted.

![\&quot;Add implementation instructions\&quot; dialog showing the link\_click 1-0-1 schema properties (elementId, elementClasses, elementTarget, targetUrl, elementContent), with targetUrl marked as required and an Edit action](/assets/images/implementation-instructions-51f52ed6ae5773cb275f0af83a076447.png)

---

# Defining the data to collect with Tracking Plans
> Create logical groupings of behavioral data by domain with explicit ownership, event specifications, and source applications to improve governance and collaboration across your organization.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/

As described in [Tracking Plans Introduction](/docs/fundamentals/tracking-plans/), a tracking plan is a logical grouping of the data you collect as an organization by domain, with an explicit owner.

With tracking plans, you can:

- Set clear ownership for the data being created
- Make tracking implementation easier
- Deliver better governance around your data
- More easily communicate what the data means and how to use it
- Collaborate more effectively with the various teams involved in delivering value from your data
- Drive a self-serve culture around data across your organization
- See event volume metrics helping to monitor data collection over time.

Each tracking plan contains event specifications that define the events you want to track. By grouping related event specifications together in tracking plans, you can better organize your data collection efforts by domain, with clear ownership and implementation instructions for each event specification.

Event specifications bridge the gap between tracking design and data collection:

- **Design phase**: you document your tracking requirements by creating event specifications that capture both technical structure and business context
- **Implementation phase**: developers use these specifications to instrument tracking code, either manually or through code generation within the Snowplow Console or using tools like Snowtype. Snowtype generated code ensures type-safety and alignment with specifications, reducing implementation errors and accelerating development time
- **Observability phase**: monitor event specification usage directly in the Console. See the total number of events collected for each specification and when each was last seen. This visibility helps you confirm implementations are live, identify unused specifications, and understand event volume patterns across your tracking plan
- **Data modeling phase**: event specifications enable automatically generated dbt models that transform atomic events into analysis-ready tables. These models understand the structure defined in your specifications, creating consistent table schemas and joining related [entities](/docs/fundamentals/entities/). As you update specifications, corresponding data models can be regenerated, keeping your warehouse transformations synchronized with your tracking design

## Elements of a Tracking Plan

**Tracking plan**

- **Name;** a descriptive and unique name for the tracking plan

- **Description;** a description of the data that the tracking plan captures

- **Owner;** the individual responsible for the tracking plan

- **Domain;** the team or business domain that owns the tracking plan

- **Source Application;** the [source application/s](/docs/event-studio/source-applications/) the Tracking Plan is implemented in

- **Event specifications**

  - **Name;** a descriptive name for the event
  - **Description;** a description to help people understand what action the event is capturing
  - **Source Applications;** the source application(s) inherited from the tracking plan containing the Application ID(s) that will be sent with this event
  - **Triggers;** specific instructions on where the event gets triggered (e.g. when a user clicks the "Add to basket" button)
  - **Event data structure;** the event data structure that this event will validate against as it is processed by your pipeline
  - **Entities;** the entities that should be attached to this event (e.g. user, product)
  - **Properties;** any specific rules for each property of the event

---

# Tracking plan templates
> Pre-defined tracking plan templates for Base Web, Base Mobile, Ecommerce, and Media tracking with included event specifications and implementation guidance.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/templates/

To create a new tracking plan based on a pre-defined template, navigate to the "Tracking plans" section from the navigation bar and click the "Create tracking plan" or "Templates" buttons.

When creating a tracking plan based on a template, the name must be unique to ensure proper identification and avoid conflicts.

> **Note:** [Snowtype](/docs/event-studio/implement-tracking/) **does not support** tracking plan templates.

Below is a list of the current templates.

## Base Web

This tracking plan template contains all the tracking of Standard Events related to web tracking. It is expected that these events would be implemented using our [Web trackers](/docs/sources/web-trackers/) and the respective [button click](/docs/sources/web-trackers/tracking-events/button-click/) and [link click](/docs/sources/web-trackers/tracking-events/link-click/) plugins.

Note: The event volume counts are calculated differently for this Tracking Plan. They are counts of any of the following events sent for the app ID's of this tracking plan.

Included event specifications:

- [Page View](/docs/sources/web-trackers/tracking-events/page-views/)
- [Page Ping](/docs/sources/web-trackers/tracking-events/activity-page-pings/)
- [Button click](/docs/sources/web-trackers/tracking-events/button-click/)
- [Link click](/docs/sources/web-trackers/tracking-events/link-click/)

## Base Mobile

This tracking plan template contains all the tracking of Standard Events related to mobile tracking. It is expected that these events would be implemented using our [mobile trackers](/docs/sources/mobile-trackers/) with [installation](/docs/sources/mobile-trackers/tracking-events/installation-tracking/), [screen view](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) and [app lifecycle](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/) tracking capabilities enabled.

Note: The event volume counts are calculated differently for this Tracking Plan. They are counts of any of the following events sent for the app ID's of this tracking plan.

Included event specifications:

- [Screen view](/docs/events/ootb-data/page-and-screen-view-events/#screen-views)
- [Screen end](/docs/events/ootb-data/page-activity-tracking/#screen-end-event)
- [Application install](/docs/events/ootb-data/mobile-lifecycle-events/#install-events)
- [Application background](/docs/events/ootb-data/mobile-lifecycle-events/#background-event)
- [Application foreground](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event)

## Ecommerce (Web and Mobile)

This Tracking Plan template contains all the basic ecommerce interactions that help you analyze customer behavior, identify potential growth opportunities, and improve your sales performance.

Tracked using the [out-of-the-box ecommerce events](/docs/events/ootb-data/ecommerce-events/) for web, Android, and iOS. On web, this requires the [Snowplow Ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/).

Included event specifications:

- Add to cart
- Checkout step
- Internal promotion click
- Internal promotion view
- Product list click
- Product list view
- Refund
- Remove from cart
- Track product view
- Transaction
- Transaction Error

## Media Web

This tracking plan template contains a full set of media tracking interactions to measure video or audio data. Tracked using the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/).

Included event specifications:

- Ad Break End Event
- Ad Break Start Event
- Ad Click Event
- Ad Complete Event
- Ad Pause Event
- Ad Quartile Event
- Ad Resume Event
- Ad Skip Event
- Ad Start Event
- Buffer End Event
- Buffer Start Event
- Error Event
- Fullscreen Change Event
- Media Ping Event
- Pause Event
- Percent Progress Event
- Picture-in-picture Change Event
- Playback End Event
- Play Event
- Quality Change Event
- Ready Event
- Seek End Event
- Seek Start Event
- Volume Change Event

---

# Anonymous tracking
> Track events without collecting user identifiers, IP addresses, or cookies to support privacy compliance and user consent requirements.
> Source: https://docs.snowplow.io/docs/events/anonymous-tracking/

Snowplow enables you to track events without collecting personally identifiable information (PII). Use anonymous tracking to respect user privacy preferences, comply with regulations such as GDPR, or track events before a user has provided consent.

Every Snowplow event includes a number of identifiers. Read more about them in the [overview](/docs/events/identifiers/) and [out-of-the-box tracking](/docs/events/ootb-data/user-and-session-identification/) pages. You can configure your trackers to prevent the collection of these identifiers, as well as IP addresses and cookies, using anonymous tracking.

Some Snowplow trackers also provide options for tracking [user consent](/docs/events/ootb-data/consent-events/) interactions and GDPR basis for processing.

## Anonymization levels

Snowplow trackers support multiple levels of anonymization to balance privacy requirements with analytics needs. Combine client-side and server-side anonymization to achieve the desired level of privacy.

As well as anonymous tracking, you can also use enrichments to pseudonymize or mask identifiers for tracked events, before they arrive at the data warehouse. This uses the [PII pseudonymization](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) or [IP anonymization](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/) enrichments.

For more details on web cookie behavior, see [Cookies and local storage](/docs/sources/web-trackers/cookies-and-local-storage/).

> **Note:** Some tracker APIs use the British English spelling: "anonymisation" rather than "anonymization".

### Full client-side anonymization

When configured for full client-side anonymization, the tracker does not track any client-side user or session identifiers. Events can still be collected, but can't be grouped by user or session.

Here's some example code for enabling full client-side anonymization when creating a tracker on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {

  // No user or session identifiers will be added to events

  // Network identifiers will be tracked

  anonymousTracking: true,

});
```

By default, the tracker uses [cookies and local storage](/docs/sources/web-trackers/cookies-and-local-storage/) to persist user and session information, as well as for buffering events. When anonymous tracking is enabled, identifiers are not saved to cookies or local storage.

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {

  TrackerConfiguration()

    .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked

    .sessionContext(false) // Session entity won't be added to events

    // Network identifiers will be tracked

}
```

**Android:**

```kotlin
createTracker(

    applicationContext,

    "appTracker",

    NetworkConfiguration("https://snowplow-collector-url.com"),

    TrackerConfiguration("appId")

        .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked

        .sessionContext(false) // Session entity won't be added to events

        // Network identifiers will be tracked

)
```

***

### Client-side anonymization with session tracking

In this mode, the tracker doesn't include user identifiers but preserves [session tracking](/docs/events/ootb-data/user-and-session-identification/). You can analyze session-level behavior without identifying individual users.

Here's some example code for enabling client-side anonymization with session tracking when creating a tracker on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {

  // User identifiers won't be added to events

  // Session and network identifiers will be tracked

  anonymousTracking: { withSessionTracking: true },

});
```

The tracker uses cookies and local storage by default. With this configuration it will store session identifiers, but not user identifiers.

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {

  TrackerConfiguration()

    .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked

    .sessionContext(true) // Session entity will be added to events but with userId as a null UUID

    // Network identifiers will be tracked

}
```

**Android:**

```kotlin
createTracker(

    applicationContext,

    "appTracker",

    NetworkConfiguration("https://snowplow-collector-url.com"),

    TrackerConfiguration("appId")

        .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked

        .sessionContext(true) // Session entity will be added to events but with userId as a null UUID

        // Network identifiers will be tracked

)
```

***

### Server-side anonymization

Server-side anonymization prevents the event [Collector](/docs/pipeline/collector/) setting the `sp` cookie for `network_userid`, or from capturing IP addresses.

Setting server-side anonymization will add a `SP-Anonymous` HTTP header to requests sent to the Collector. This requires the request method to be `POST`, which is the default for most trackers. When the `SP-Anonymous` header is present, the Collector doesn't set or read the `sp` cookie.

An alternative method for preventing IP address tracking is to set a null value, such as `0.0.0.0` within the tracker. Most Snowplow trackers [support this option](/docs/events/ootb-data/user-and-session-identification/), although not the JavaScript trackers.

In the [JavaScript trackers](/docs/sources/web-trackers/anonymous-tracking/), setting server-side anonymization also sets full client-side anonymization. In the [native mobile trackers](/docs/sources/mobile-trackers/anonymous-tracking/), it's a separate configuration. You can choose either client-side anonymization, server-side anonymization, or both.

Here's some example code for enabling server-side anonymization when creating a tracker on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {

  // No user, session, or network identifiers will be tracked

  anonymousTracking: { withServerAnonymisation: true },

});
```

For fully [cookieless](/docs/sources/web-trackers/cookies-and-local-storage/) web tracking, configure the tracker with:

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {

  // No user, session, or network identifiers will be tracked

  // No data will be stored in the browser

  anonymousTracking: { withServerAnonymisation: true },

  stateStorageStrategy: 'none',

});
```

This configuration prevents the tracker from storing any data in cookies or local storage.

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {

  // User and session identifiers will be tracked

  // Network identifiers won't be added to events

  EmitterConfiguration().serverAnonymisation(true)

}
```

**Android:**

```kotlin
createTracker(

    applicationContext,

    "appTracker",

    NetworkConfiguration("https://snowplow-collector-url.com"),

    // User and session identifiers will be tracked

    // Network identifiers won't be added to events

    EmitterConfiguration().serverAnonymisation(true)

)
```

***

## Tracker support

This table shows the support for anonymous tracking across the main [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                                    | Supported | Since version | Client-side | Server-side | Notes                                                         |
| ------------------------------------------------------------------------------------------ | --------- | ------------- | ----------- | ----------- | ------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/anonymous-tracking/)                                      | ✅         | 3.0.0         | ✅           | ✅           |                                                               |
| [iOS](/docs/sources/mobile-trackers/anonymous-tracking/)                                   | ✅         | 4.0.0         | ✅           | ✅           |                                                               |
| [Android](/docs/sources/mobile-trackers/anonymous-tracking/)                               | ✅         | 4.0.0         | ✅           | ✅           |                                                               |
| [React Native](/docs/sources/react-native-tracker/anonymous-tracking/)                     | ❌/✅       | 1.3.0         | ❌/✅         | ✅           | Version 4.x only supports server-side anonymization           |
| [Flutter](/docs/sources/flutter-tracker/anonymous-tracking/)                               | ✅         | 0.3.0         | ✅           | ✅           |                                                               |
| [Roku](/docs/sources/roku-tracker/adding-data/#adding-user-and-platform-data-with-subject) | ✅         | 0.3.0         | ✅           | ✅           | Use Subject configuration to manage client-side anonymization |
| [Node.js](/docs/sources/node-js-tracker/initialization/)                                   | ✅         | 3.21.0        | ❌           | ✅           |                                                               |
| Golang                                                                                     | ❌         |               |             |             |                                                               |
| .NET                                                                                       | ❌         |               |             |             |                                                               |
| Java                                                                                       | ❌         |               |             |             |                                                               |
| Python                                                                                     | ❌         |               |             |             |                                                               |
| Scala                                                                                      | ❌         |               |             |             |                                                               |
| Ruby                                                                                       | ❌         |               |             |             |                                                               |
| Rust                                                                                       | ❌         |               |             |             |                                                               |
| [PHP](/docs/sources/php-tracker/emitters/)                                                 | ✅         | 0.9.0         | ❌           | ✅           |                                                               |
| C++                                                                                        | ❌         |               |             |             |                                                               |
| Unity                                                                                      | ✅         | 0.9.0         | ✅           | ✅           |                                                               |
| Lua                                                                                        | ❌         |               |             |             |                                                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)                  | ✅         | v4            | ✅           | ✅           |                                                               |

### Toggle anonymous tracking

As well as setting it at tracker initialization, you can enable or disable anonymous tracking during a session. Use this to start with anonymous tracking until a user accepts a cookie banner, or to enable full tracking when a user logs in.

Here's some example code for toggling anonymization at runtime on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('enableAnonymousTracking');

snowplow('disableAnonymousTracking');
```

These methods also accept an [optional configuration object](/docs/sources/web-trackers/anonymous-tracking/#toggling-anonymous-tracking) to specify which levels of anonymization to enable.

**iOS:**

```swift
tracker.userAnonymisation = true

tracker.sessionContext = true

tracker.emitter?.serverAnonymisation = true
```

**Android:**

```kotlin
tracker.userAnonymisation = true

tracker.sessionContext = true

tracker.emitter.serverAnonymisation = true
```

***

On web, enabling anonymous tracking removes user identifiers from outgoing events but doesn't delete existing cookies. On mobile, toggling anonymization starts a new session.

### Clear user data

When enabling anonymous tracking, you may want to clear any existing user identifiers stored in cookies or local storage.

Use the `clearUserData()` method in the JavaScript trackers to delete all data stored in the `_sp_id` and `_sp_ses` cookies. It takes an [optional configuration object](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) to specify which identifiers to clear. To delete the `network_userid` in the Collector `sp` cookie, set server-side anonymization.

There's no equivalent API for deleting stored data on mobile.

## Anonymizable event properties

Use anonymous tracking to avoid collecting the following identifiers in your events. The table shows how you can anonymize each identifier using client-side anonymization, server-side anonymization, or enrichments.

> **Note:** Snowplow events have two properties with similar names: the `user_id` atomic event property and the `userId` property in the session entity.
>
> The former is a business user identifier you set via the tracker API, while the latter is a device ID generated by the tracker. Both can be anonymized using client-side anonymization.

| Property                   | Location in event                                                   | Identifier type | Description                                                 | Client-side anon | Server-side anon | Enrichment                                                                                                                                                                                    |
| -------------------------- | ------------------------------------------------------------------- | --------------- | ----------------------------------------------------------- | ---------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `domain_userid`            | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | User            | UUID stored in a first-party cookie                         | ✅                |                  | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `domain_sessionid`         | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Session         | UUID for the current session                                | ✅                |                  | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `domain_sessionidx`        | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Session         | Count of sessions for this user                             | ✅                |                  |                                                                                                                                                                                               |
| `network_userid`           | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Network         | UUID set by the Collector in a server-side cookie           |                  | ✅                | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `user_ipaddress`           | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Network         | IP address captured by the Collector                        |                  | ✅                | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/), [IP anonymization](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/) |
| `user_id` in atomic fields | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | User            | Business user identifier set via `setUserId` or similar API | ✅                |                  | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `userId` in session entity | [Entity](/docs/events/ootb-data/user-and-session-identification/)   | User            | Device ID in the client session entity                      | ✅                |                  |                                                                                                                                                                                               |
| `sessionId`                | [Entity](/docs/events/ootb-data/user-and-session-identification/)   | Session         | UUID for the session                                        | ✅                |                  |                                                                                                                                                                                               |
| `appleIdfa`                | [Entity](/docs/events/ootb-data/device-and-browser/#mobile-entity)  | User            | Advertising identifier (IDFA) on iOS                        | ✅                |                  |                                                                                                                                                                                               |
| `appleIdfv`                | [Entity](/docs/events/ootb-data/device-and-browser/#mobile-entity)  | User            | Vendor identifier (IDFV) on iOS                             | ✅                |                  |                                                                                                                                                                                               |
| `androidIdfa`              | [Entity](/docs/events/ootb-data/device-and-browser/#mobile-entity)  | User            | Advertising identifier on Android                           | ✅                |                  |                                                                                                                                                                                               |
| `tabId`                    | [Entity](/docs/events/ootb-data/device-and-browser/#browser-entity) | User            | UUID for the browser tab                                    | ✅                |                  |                                                                                                                                                                                               |

The [PII pseudonymization enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) supports an `anonymousOnly` mode, available since Enrich version 5.3.0, that only pseudonymizes fields when the `SP-Anonymous` header is present. This lets you keep full identifiers for consented users while anonymizing non-consented users.

It can also pseudonymize fields for [structured](/docs/fundamentals/canonical-event/#structured-events) and [legacy ecommerce](/docs/fundamentals/canonical-event/#legacy-ecommerce-fields) events, as well as these fields that are populated during enrichment:

| Property             | Location in event                                                          | Identifier type | Description                                                                                                                   |
| -------------------- | -------------------------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `refr_domain_userid` | [Atomic](/docs/fundamentals/canonical-event/#cross-domain-tracking-fields) | User            | From [cross-domain tracking](/docs/events/cross-navigation/)                                                                  |
| `ip_organization`    | [Atomic](/docs/fundamentals/canonical-event/#ip-address-fields)            | User            | Added by [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)                              |
| `ip_domain`          | [Atomic](/docs/fundamentals/canonical-event/#ip-address-fields)            | User            | Added by [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)                              |
| `mkt_term`           | [Atomic](/docs/fundamentals/canonical-event/#marketing-fields)             | Marketing       | Added by [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) |
| `mkt_content`        | [Atomic](/docs/fundamentals/canonical-event/#marketing-fields)             | Marketing       | Added by [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) |
| `mkt_clickid`        | [Atomic](/docs/fundamentals/canonical-event/#marketing-fields)             | Marketing       | Added by [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) |

---

# Cross-navigation tracking
> Configure cross-domain tracking to maintain user identity between different domains in your ecosystem
> Source: https://docs.snowplow.io/docs/events/cross-navigation/

When users navigate between different domains in your ecosystem, their user identity may get fragmented. Examples include navigation from your main website to a partner site, or to a mobile app. This creates gaps in your user journey data and makes it difficult to understand the complete customer experience across your digital properties.

> **Info:** This problem doesn't usually apply for navigation between web subdomains, for example between `www.example.com` and `blog.example.com`. This is because the web trackers have `discoverRootDomain` [enabled by default](/docs/sources/web-trackers/tracker-setup/initialization-options/).

Cross-navigation (also called cross-domain) tracking solves this problem by passing user identification data in URL parameters when users click links to other domains. This enables you to maintain user continuity across your applications.

To use cross-navigation tracking, configure the [web](/docs/sources/web-trackers/cross-domain-tracking/) or [native mobile](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) trackers to add an additional parameter, named `_sp`, to the querystring of outbound links. This process is called "link decoration".

Link decoration makes the added values visible in the `url` field of events on the destination page or URI. The querystring is added only to the events at the destination page: it doesn't persist throughout the user's session.

## Querystring properties

The `_sp` querystring parameter has two different formats: extended or short. You can also configure exactly which properties you want to include.

Available properties:

| Property         | Description                                    | Extended | Short |
| ---------------- | ---------------------------------------------- | -------- | ----- |
| `domainUserId`   | Current tracker-generated UUID user identifier | ✅        | ✅     |
| `timestamp`      | Current epoch timestamp, millisecond precision | ✅        | ✅     |
| `sessionId`      | Current session UUID identifier                | ✅        |       |
| `subjectUserId`  | Custom business user identifier                | ✅        |       |
| `sourceId`       | Application identifier                         | ✅        |       |
| `sourcePlatform` | Platform of the current device                 | ✅        |       |
| `reason`         | Custom information or identifier               | ✅        |       |

For example, the link `appSchema://path/to/page` would look like this after decoration with the full extended format:

```text
appSchema://path/to/page?_sp=domainUserId.timestamp.sessionId.subjectUserId.sourceId.sourcePlatform.reason
```

## How are the querystring parameters processed?

By default, your pipeline will process only the short format querystring `_sp={domainUserId}.{timestamp}`, even if you've decorated the links with the extended format. To process the extra properties, you'll need to configure the [cross-navigation enrichment](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/).

Both default and extended formats will populate the [atomic](/docs/fundamentals/canonical-event/) `refr_domain_userid` and `refr_dvce_tstamp` fields. The enrichment also adds a `cross_navigation` entity to the event.

| Feature                                                 | Default behavior | With cross-navigation enrichment |
| ------------------------------------------------------- | ---------------- | -------------------------------- |
| Processes short format `_sp={domainUserId}.{timestamp}` | ✅                | ✅                                |
| Processes extended format properties                    | ❌                | ✅                                |
| Populates `refr_domain_userid` field                    | ✅                | ✅                                |
| Populates `refr_dvce_tstamp` field                      | ✅                | ✅                                |
| Adds `cross_navigation` entity                          | ❌                | ✅                                |

## Tracker support

This table shows the support for cross-domain tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include link click tracking.

| Tracker                                                                                                                              | Supported | Since version |
| ------------------------------------------------------------------------------------------------------------------------------------ | --------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/link-click/)                                                                        | ✅         | 0.7.0         |
| [iOS](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking)     | ✅         | 6.0.0         |
| [Android](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) | ✅         | 6.0.0         |
| React Native                                                                                                                         | ❌         |               |
| Flutter                                                                                                                              | ❌         |               |

## Identities and cross-navigation tracking

[Snowplow Identities](/docs/identities/) provides real-time identity resolution, stitching together user identifiers from different events into unified profiles.

Cross-navigation tracking is directly supported within Identities. If you enable [cross-domain tracking support](/docs/identities/concepts/cross-domain-tracking/), Identities will automatically treat `refr_domain_userid` as equivalent to `domain_userid`. The result is that all of a user's activity across your domains — anonymous or authenticated — resolves to the same `snowplow_id`, without any downstream modeling to join the identifiers yourself.

---

# How to track custom events and entities
> Track custom self-describing events and entities with JSON schemas for flexible data collection.
> Source: https://docs.snowplow.io/docs/events/custom-events/

Snowplow includes three ways to customize tracking:

- Custom [self-describing events](/docs/fundamentals/events/#self-describing-events)
- Custom [entities](/docs/fundamentals/entities/) added to out-of-the-box or custom events
- Structured events (not recommended)

Read our [tracking design best practice guide](/docs/fundamentals/tracking-design-best-practice/) to learn how to track and use custom data.

## Custom events

Self-describing [events](/docs/fundamentals/events/#self-describing-events) are [based on JSON schemas](/docs/fundamentals/schemas/) and can have arbitrarily many fields.

To define your own custom event, you will need to [create a corresponding schema](/docs/fundamentals/schemas/). Snowplow uses the schema to validate that the JSON containing the event properties is well-formed.

This code shows how you could track a custom `article_share` event using the [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
window.snowplow('trackSelfDescribingEvent', {

  event: {

    schema: 'iglu:com.example_company/article_share/jsonschema/1-0-0',

    data: {

      articleId: 'doc-12345',

      shareMethod: 'email',

      articleTitle: 'Getting Started with Snowplow'

    }

  }

});
```

In addition to populating the [standard atomic columns](/docs/fundamentals/canonical-event/), each type of self-describing event gets its own warehouse column (or its own table, in the case of Redshift) for event-specific fields defined in its schema. See the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page for more information.

## Custom entities

As with custom self-describing events, if you want to create your own custom [entity](/docs/fundamentals/entities/), you will need to [create a corresponding schema](/docs/fundamentals/schemas/). Snowplow uses the schema to validate that the JSON containing the entity properties is well-formed.

Here's an example that shows how you could track custom `user` and `product` entities along with a page view event, using the [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
// Track a page view with a custom entity

window.snowplow('trackPageView', {

  context: [

    {

      schema: 'iglu:com.example_company/user/jsonschema/1-0-0',

      data: {

        userId: 'user-12345',

        accountType: 'enterprise',

        subscriptionTier: 'premium'

      }

    },

    {

      schema: 'iglu:com.example_company/product/jsonschema/1-2-1',

      data: {

        productId: 'ASO01043',

        brand: 'ACME'

      }

    }]

});
```

See the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page to learn how entity data is structured in the data warehouse.

> **Info:** In the past, what we now call "entity" or "entities" was called "context". You'll still find `context` used in many of the existing APIs, database column names, and documentation, especially to refer to a set of multiple entities. For example, the `context` parameter in the JavaScript tracker API above is an array of entities.

### Add custom entities to all events

Certain Snowplow trackers provide the option to add custom entities to all events, or a configurable subset of events. These are called **application entities**. This feature is called "global context" in the trackers.

See the documentation for each tracker to learn how to configure it:

- [Web](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/)
- [Native mobile](/docs/sources/mobile-trackers/custom-tracking-using-schemas/global-context/) (iOS and Android)
- [React Native](/docs/sources/react-native-tracker/)
- [Scala](/docs/sources/scala-tracker/initialization/)

Use [source applications](/docs/event-studio/source-applications/) to document your expected application entities.

## Structured events

> **Info:** Structured event tracking is a legacy format used to track events that weren't natively supported by Snowplow.
>
> We recommend using self-describing events for custom event tracking.

Structured events are simpler to create than custom self-describing events, as you don't need to define a [schema](/docs/fundamentals/schemas/). However, they have a number of disadvantages:

|            | Structured events                           | Self-describing events                |
| ---------- | ------------------------------------------- | ------------------------------------- |
| Format     | ❌ Data must fit the fields below            | ✅ JSON, as complex as you want        |
| Validation | ❌ No validation (beyond field types)        | ✅ Schema includes validation criteria |
| Meaning    | ❌ Can only infer what each field represents | ✅ Schema includes field descriptions  |

Structured events have five custom event specific parameters:

| **Table column** | **Type** | **Description**                                                        | **Example values**            |
| ---------------- | -------- | ---------------------------------------------------------------------- | ----------------------------- |
| `se_category`    | text     | The category of event                                                  | `Ecomm`, `Media`              |
| `se_action`      | text     | The action / event itself                                              | `add-to-basket`, `play-video` |
| `se_label`       | text     | A label often used to refer to the 'object' the action is performed on | `dog-skateboarding-video`     |
| `se_property`    | text     | A property associated with either the action or the object             | `hd`                          |
| `se_value`       | decimal  | A value associated with the user action                                | `13.99`                       |

Here's how to track a structured event using the [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
snowplow('trackStructEvent', {

  category: 'Product',

  action: 'View',

  label: 'ASO01043',

  property: 'Dress',

  value: 49.95

});
```

In the warehouse, data tracked for any of these structured event-specific fields is stored in [standard atomic columns](/docs/fundamentals/canonical-event/#structured-events).

---

# Filtering bot events
> Detect and filter out bot traffic to keep your analytics accurate and your data warehouse or lake clean.
> Source: https://docs.snowplow.io/docs/events/filtering-bot-events/

Bot traffic — crawlers, scrapers, and automated agents — can make up a significant share of the events flowing through your pipeline.

If left unfiltered, these events inflate page view counts, distort session metrics, and degrade the quality of any downstream analysis or machine learning model that relies on your data.

Filtering bot events helps you maintain accurate analytics and can reduce unnecessary storage and compute costs in your data warehouse.

## Detect bot traffic

Snowplow can identify bot traffic both client-side, via a [tracker plugin](/docs/sources/web-trackers/tracking-events/bot-detection/), and server-side, via [enrichments](/docs/pipeline/enrichments/).

Each source attaches information to the event that you can use for filtering. For convenience, the [bot detection enrichment](#bot-detection-enrichment) consolidates all sources into a single `bot` field.

> **Note:** Events tracked with [server-side anonymization](/docs/events/anonymous-tracking/#server-side-anonymization) lack IP address data. As such, bot detection based on IP addresses will not be effective for these events. This includes the ASN lookup enrichment and the IP-related check in the IAB enrichment.

### Client-side bot detection plugin

The [bot detection plugin](/docs/sources/web-trackers/tracking-events/bot-detection/) for the web tracker runs in the browser to detect automation frameworks, such as Selenium, PhantomJS, or headless Chrome, using behavioral signals. It attaches a `client_side_bot_detection` [entity](/docs/fundamentals/entities/) with a `bot` boolean and a `kind` field identifying the bot type.

### IAB enrichment

The [IAB enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) checks the event's IP address and user agent string against the [IAB/ABC International Spiders and Bots List](https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/), an industry-standard database maintained by the Interactive Advertising Bureau. It adds an [entity](/docs/fundamentals/entities/) with a `spiderOrRobot` boolean that indicates whether the event came from a known bot.

> **Tip:** When configuring this enrichment, you can also [add custom user agent string patterns](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/#custom-user-agent-lists) you'd like to classify as bots or not bots.

### YAUAA enrichment

The [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) performs advanced user agent parsing using the [YAUAA library](https://yauaa.basjes.nl/). Among many other fields, it classifies the device and agent type — values such as `"Robot"`, `"Robot Mobile"`, or `"Robot Imitator"` in the `deviceClass` or `agentClass` fields indicate bot traffic.

### ASN lookup enrichment

The [ASN lookup enrichment](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/) checks the event's autonomous system number against a configurable list of ASNs associated with bots, cloud providers, or data centers. Many bots originate from well-known hosting ASNs, and community-maintained lists track these. When a match is found, the enrichment sets `likelyBot` to `true` on the ASN entity.

> **Tip:** For this enrichment to work, you also need to enable the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) and configure either its `asn` or `isp` setting, as this is what will generate the initial ASN entity.

### Bot detection enrichment

The [bot detection enrichment](/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/) consolidates the outputs of the sources above (including the client-side bot detection plugin and the server-side enrichments) into a single `bot_detection` entity. It uses "any positive = bot" logic: if any enabled source indicates that the event is a bot, the `bot` field is set to `true`, along with a list of which sources contributed to the decision. This gives you a single field to check instead of querying multiple entities separately.

## Filter out bot events

Once you have bot detection in place, there are two main approaches to removing bot events from your analytics.

### Filter events in the warehouse

If you prefer to keep all events in your warehouse or lake and filter at query time or in your data models, use the `bot_detection` entity added by the [bot detection enrichment](/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/). For example, in SQL:

```sql
select *

from events

where not contexts_com_snowplowanalytics_snowplow_bot_detection_1[0]:bot::boolean
```

This approach lets you keep the bot events for inspection or debugging if needed.

### Drop events in the pipeline

If you want to prevent bot events from reaching your warehouse or lake entirely, use a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/) to drop them during processing.

The bot detection enrichment runs before the JavaScript enrichment, so you can read its output in your JavaScript code. Use `event.getDerived_contexts()` to access the `bot_detection` entity and drop the event if it is flagged as a bot:

```js
function process(event) {

  const entities = JSON.parse(event.getDerived_contexts());

  if (entities) {

    for (const entity of entities.data) {

      if (entity.schema.startsWith('iglu:com.snowplowanalytics.snowplow/bot_detection/jsonschema/1') &&

          entity.data.bot) {

        event.drop();

      }

    }

  }

}
```

The `event.drop()` method (available since Enrich 5.3.0) prevents the event from being sent to any stream or destination, lowering infrastructure costs.

> **Warning:** There is no way to recover dropped events, so use this with caution.

An alternative is to throw an exception, which sends the event to [failed events](/docs/fundamentals/failed-events/) instead of dropping it entirely:

```js
function process(event) {

  const entities = JSON.parse(event.getDerived_contexts());

  if (entities) {

    for (const entity of entities.data) {

      if (entity.schema.startsWith('iglu:com.snowplowanalytics.snowplow/bot_detection/jsonschema/1') &&

          entity.data.bot) {

        throw "Filtered a bot event";

      }

    }

  }

}
```

> **Note:** This creates an "enrichment failure" failed event, which may be tricky to distinguish from genuine failures in your enrichment code.

---

# Example tracker HTTP request headers
> Reference examples of HTTP requests to the Snowplow Collector using the Tracker Protocol with JSON payloads.
> Source: https://docs.snowplow.io/docs/events/http-requests/

This page gives examples of requests to the Snowplow [Collector](/docs/pipeline/collector/). This is a reference **for those who plan to implement their own tracker**, or want to understand the requests being sent by existing Snowplow trackers.

Below are a number of example [Tracker Protocol](/docs/fundamentals/canonical-event/) requests. All examples are POST requests with a JSON body. Requests to the Collector endpoint can contain more than one event, depending how your tracker is configured.

All requests are sent to `https://<your-collector-host>/com.snowplowanalytics.snowplow/tp2` unless you have specified a custom POST path when [configuring your Collector](/docs/pipeline/collector/).

All payloads should be wrapped in a `payload_data` self-describing JSON.

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      ...

    }

  ]

}
```

> **Info:** Reserved parameter `u``u` is a reserved parameter because it is used for [click tracking in the Pixel Tracker](/docs/sources/pixel-tracker/#click-tracking).

## Self-describing event

This is an example of an HTTP request for a `viewed_product` event from web, using the recommended Base64 encoding and the `ue_px` property:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      "e": "ue",

      "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",

      "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",

      "tv": "js-3.5.0",

      "tna": "biz1",

      "aid": "website",

      "p": "web",

      "cookie": "1",

      "cs": "UTF-8",

      "lang": "en-GB",

      "res": "3440x1440",

      "cd": "30",

      "tz": "Europe/London",

      "dtm": "1665398554151",

      "vp": "693x1302",

      "ds": "678x9015",

      "vid": "29",

      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",

      "duid": "85094061-f702-4b62-a46d-20f7226b4741",

      "url": "https://snowplow.io/",

      "stm": "1665398554153"

    }

  ]

}
```

The encoded base64 string (`ue_px`):

```text
eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==
```

represents the following JSON, when decoded:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",

  "data": {

    "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",

    "data": {

      "product_id": "ASO01043",

      "price": 49.95

    }

  }

}
```

**cURL example**

```bash
curl --request POST \

    --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \

    --header 'Content-Type: application/json' \

    --data '{

    "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

    "data": [

      {

        "e": "ue",

        "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",

        "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",

        "tv": "js-3.5.0",

        "tna": "biz1",

        "aid": "website",

        "p": "web",

        "cookie": "1",

        "cs": "UTF-8",

        "lang": "en-GB",

        "res": "3440x1440",

        "cd": "30",

        "tz": "Europe/London",

        "dtm": "1665398554151",

        "vp": "693x1302",

        "ds": "678x9015",

        "vid": "29",

        "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",

        "duid": "85094061-f702-4b62-a46d-20f7226b4741",

        "url": "https://snowplow.io/",

        "stm": "1665398554153"

      }

    ]

  }'
```

## Custom entities

An array of [entities](/docs/fundamentals/entities/) can be sent with each event. Entity payloads should be wrapped in a `contexts` self-describing JSON.

This is an example of an HTTP request for the same `viewed_product` event as above but with a custom entity added, using the recommended Base64 encoding and the `cx` property:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      "e": "ue",

      "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",

      "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",

      "tv": "js-3.5.0",

      "tna": "biz1",

      "aid": "website",

      "p": "web",

      "cookie": "1",

      "cs": "UTF-8",

      "lang": "en-GB",

      "res": "3440x1440",

      "cd": "30",

      "tz": "Europe/London",

      "dtm": "1665398554151",

      "vp": "693x1302",

      "ds": "678x9015",

      "vid": "29",

      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",

      "duid": "85094061-f702-4b62-a46d-20f7226b4741",

      "url": "https://snowplow.io/",

      "stm": "1665398554153",

      "cx": "ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29udGV4dHMvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiBbCiAgICB7CiAgICAgICJzY2hlbWEiOiAiaWdsdTpjb20ubXlfY29tcGFueS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiZGF0YSI6IHsKICAgICAgICAibXlfY29tcGFueV91c2VyX2lkIjogIjk5OTl4eXoiCiAgICAgIH0KICAgIH0KICBdCn0="

    }

  ]

}
```

The encoded base64 string (`cx`):

```text
ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29udGV4dHMvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiBbCiAgICB7CiAgICAgICJzY2hlbWEiOiAiaWdsdTpjb20ubXlfY29tcGFueS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiZGF0YSI6IHsKICAgICAgICAibXlfY29tcGFueV91c2VyX2lkIjogIjk5OTl4eXoiCiAgICAgIH0KICAgIH0KICBdCn0=
```

represents the following JSON, when decoded:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",

  "data": [

    {

      "schema": "iglu:com.my_company/user/jsonschema/1-0-0",

      "data": {

        "my_company_user_id": "9999xyz"

      }

    }

  ]

}
```

**cURL example**

```bash
curl --request POST \

   --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \

   --header 'Content-Type: application/json' \

   --data '{

   "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

   "data": [

     {

       "e": "ue",

       "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",

       "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",

       "tv": "js-3.5.0",

       "tna": "biz1",

       "aid": "website",

       "p": "web",

       "cookie": "1",

       "cs": "UTF-8",

       "lang": "en-GB",

       "res": "3440x1440",

       "cd": "30",

       "tz": "Europe/London",

       "dtm": "1665398554151",

       "vp": "693x1302",

       "ds": "678x9015",

       "vid": "29",

       "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",

       "duid": "85094061-f702-4b62-a46d-20f7226b4741",

       "url": "https://snowplow.io/",

       "stm": "1665398554153",

       "cx": "ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29udGV4dHMvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiBbCiAgICB7CiAgICAgICJzY2hlbWEiOiAiaWdsdTpjb20ubXlfY29tcGFueS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiZGF0YSI6IHsKICAgICAgICAibXlfY29tcGFueV91c2VyX2lkIjogIjk5OTl4eXoiCiAgICAgIH0KICAgIH0KICBdCn0="

     }

   ]

 }'
```

## Page view

This is an example HTTP request for a [page view](/docs/events/ootb-data/page-activity-tracking/) event:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      "e": "pv",

      "url": "https://snowplow.io/",

      "page": "Snowplow: behavioral data creation leader",

      "eid": "2ec13853-ef9e-414a-8976-e64a4693b134",

      "tv": "js-3.5.0",

      "tna": "biz1",

      "aid": "website",

      "p": "web",

      "cookie": "1",

      "cs": "UTF-8",

      "lang": "en-GB",

      "res": "3440x1440",

      "cd": "30",

      "tz": "Europe/London",

      "dtm": "1665398554084",

      "vp": "693x1302",

      "ds": "678x8188",

      "vid": "29",

      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",

      "duid": "85094061-f702-4b62-a46d-20f7226b4741",

      "stm": "1665398554084"

    }

  ]

}
```

**cURL example**

```bash
curl --request POST \

  --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \

  --header 'Content-Type: application/json' \

  --data '{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      "e": "pv",

      "url": "https://snowplow.io/",

      "page": "Snowplow: behavioral data creation leader",

      "eid": "2ec13853-ef9e-414a-8976-e64a4693b134",

      "tv": "js-3.5.0",

      "tna": "biz1",

      "aid": "website",

      "p": "web",

      "cookie": "1",

      "cs": "UTF-8",

      "lang": "en-GB",

      "res": "3440x1440",

      "cd": "30",

      "tz": "Europe/London",

      "dtm": "1665398554084",

      "vp": "693x1302",

      "ds": "678x8188",

      "vid": "29",

      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",

      "duid": "85094061-f702-4b62-a46d-20f7226b4741",

      "stm": "1665398554084"

    }

  ]

}'
```

## Page ping

This is an example HTTP request for a [page ping](/docs/events/ootb-data/page-activity-tracking/) event:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      "e": "pp",

      "url": "https://snowplow.io/",

      "page": "Snowplow: behavioral data creation leader",

      "eid": "0a88e2bf-a2cd-48e7-877b-608c4b3d32a6",

      "tv": "js-3.5.0",

      "tna": "snplow5",

      "aid": "website",

      "p": "web",

      "cookie": "1",

      "cs": "UTF-8",

      "lang": "en-GB",

      "res": "3440x1440",

      "cd": "30",

      "tz": "Europe/London",

      "dtm": "1665401016632",

      "cx": "",

      "vp": "693x1302",

      "ds": "678x9015",

      "vid": "52",

      "sid": "6a3769de-da79-431d-85dd-0a1b96e2f7fd",

      "duid": "1f08df65-1558-46fc-b586-838460fc438e",

      "stm": "1665401016633"

    }

  ]

}
```

**cURL example**

```bash
curl --request POST \

  --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \

  --header 'Content-Type: application/json' \

  --data '{

  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

  "data": [

    {

      "e": "pp",

      "url": "https://snowplow.io/",

      "page": "Snowplow: behavioral data creation leader",

      "eid": "0a88e2bf-a2cd-48e7-877b-608c4b3d32a6",

      "tv": "js-3.5.0",

      "tna": "snplow5",

      "aid": "website",

      "p": "web",

      "cookie": "1",

      "cs": "UTF-8",

      "lang": "en-GB",

      "res": "3440x1440",

      "cd": "30",

      "tz": "Europe/London",

      "dtm": "1665401016632",

      "cx": "",

      "vp": "693x1302",

      "ds": "678x9015",

      "vid": "52",

      "sid": "6a3769de-da79-431d-85dd-0a1b96e2f7fd",

      "duid": "1f08df65-1558-46fc-b586-838460fc438e",

      "stm": "1665401016633"

    }

  ]

}'
```

---

# User and session identifiers
> Track user and session identifiers including domain_userid, network_userid, session IDs, business user IDs, and the Snowplow ID from Identities.
> Source: https://docs.snowplow.io/docs/events/identifiers/

Snowplow tracks several identifiers to help you understand user behavior across sessions and devices. These include tracker-generated IDs stored on the client, server-generated IDs from the Collector, and business user IDs you set in your app.

Understanding how these identifiers work enables you to build a complete picture of user journeys, whether through real-time identity resolution with [Snowplow Identities](/docs/identities/), or identity stitching in your data models.

Read more about tracking identifiers in the [OOTB user and session data](/docs/events/ootb-data/user-and-session-identification/) page.

## User identifiers

For identifiers in cookies, consider using the [Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) to prevent identifier loss due to browser restrictions.

### Tracker-generated

This on-device user identifier is **generated by the tracker**. It takes the form of a UUID, and is [tracked differently](/docs/events/ootb-data/user-and-session-identification/) on web or mobile.

It's the default identifier used in our [dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) to identify users. The [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) package coalesces the `domain_userid` and the `userId` property into a single `user_identifier` field, to make it easier to work with. The value can come from either place.

|                       | Web                                                          | Mobile                    |
| --------------------- | ------------------------------------------------------------ | ------------------------- |
| Event field           | `domain_userid`                                              | `userId`                  |
| Location in event     | [Atomic event property](/docs/fundamentals/canonical-event/) | Client session entity     |
| Persistance mechanism | Browser cookies                                              | App user settings storage |
| Unified Digital field | `user_identifier`                                            | `user_identifier`         |

> **Note:** In mobile apps, there are additional on-device identifiers provided by the platform: advertising ID (IDFA) and vendor ID (IDFV, app set ID). These can be tracked in the [mobile entity](/docs/events/ootb-data/device-and-browser/#mobile-entity).

### Server-generated

The Snowplow Collector generates the `network_userid` user identifier. It's a UUID that's returned in the Collector's HTTP response to the tracker, as the `sp` cookie in a `Set-Cookie` header.

Whether the `network_userid` persists between events or sessions depends on the tracker:

| Tracker                                                                      | Persists within session | Persists across sessions | Mechanism                                                                                                     |
| ---------------------------------------------------------------------------- | ----------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#business-user-id)          | ✅                       | ✅                        | Stored in `sp` cookie                                                                                         |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅                       | ✅                        | Stored using `HTTPCookieStorage`                                                                              |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅                       | ✅                        | Stored using `SharedPreferences`                                                                              |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅                       | ✅                        | Depends on platform used                                                                                      |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅                       | ✅                        | Depends on platform used                                                                                      |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ❌                       | ❌                        |                                                                                                               |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ❌                       | ❌                        |                                                                                                               |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ❌                       | ❌                        |                                                                                                               |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ❌                       | ❌                        |                                                                                                               |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅                       | ❌/✅                      | Stored in memory by default; configure a `CollectorCookieJar` to persist the `network_userid` across sessions |
| [Python](/docs/sources/python-tracker/subject/)                              | ❌                       | ❌                        |                                                                                                               |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ❌                       | ❌                        |                                                                                                               |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ❌                       | ❌                        |                                                                                                               |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ❌                       | ❌                        |                                                                                                               |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ❌                       | ❌                        |                                                                                                               |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅                       | ❌/✅                      | Stored in memory by default; configure a `cookie_file` to persist the `network_userid` across sessions        |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ❌                       | ❌                        | Set all properties.                                                                                           |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                   | ❌                       | ❌                        | Set `user_id` only.                                                                                           |
| Google Tag Manager                                                           | ✅                       | ✅                        | Same as for the web tracker: stored in `sp` cookie                                                            |

Some trackers also allow you to [set your own](/docs/events/ootb-data/user-and-session-identification/) `network_userid` value.

### Business-generated

The business-generated user identifier is an external identifier given to the tracker by the application. Most commonly it's the username or email address of the logged in user e.g. `jane.doe@email.com`.

It's tracked as the atomic `user_id` field. All Snowplow trackers have an API to [set this identifier](/docs/events/ootb-data/user-and-session-identification/).

### Snowplow ID

If you use [Snowplow Identities](/docs/identities/), each event will receive a Snowplow ID from the resolved user profile. The ID is added in an [identity entity](/docs/identities/concepts/#identity-entity). You can configure Identities to stitch together any identifiers for identity resolution.

### IP address

Snowplow events also contain the IP address of the user. The value is stored in the [atomic](/docs/fundamentals/canonical-event/) `user_ipaddress` property.

We don't recommend using the IP address as a primary identifier: multiple users can share an IP address, and a user's IP address can change frequently. However, it can be useful for certain use cases such as bot detection or geolocation.

## Session identifiers

The main Snowplow session identifiers are the session ID and session index:

- Session ID: a UUID generated by the tracker at the start of the session
- Session index: the index of the session for the current user, based on the `domain_userid` or `userId` tracker-generated user identifier

They're tracked differently depending on the tracker. On web, they're tracked as the `domain_sessionid` and `domain_sessionidx` [atomic event](/docs/fundamentals/canonical-event/) fields. In other trackers, they're part of the client session entity, which also includes other session-related information. Note that the web trackers can also track the client session entity, but not all trackers support [session tracking](/docs/events/ootb-data/user-and-session-identification/).

## Identity resolution

Identity resolution is the process of taking various user identifiers and combining them into a single user identifier, to better identify and track users throughout their journey on your site or app. It allows you to attribute logged-in and non-logged-in sessions back to a single user.

Snowplow offers two approaches to identity resolution:

- **[Snowplow Identities](/docs/identities/)** resolves identity in real time during enrichment, attaching a unified Snowplow ID to each event. The [Identities dbt package](/docs/identities/data-models/) then builds mapping tables for use in downstream models.
- **[Identity stitching in the Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/)** resolves identity at modeling time by stitching the business user ID with tracker-generated user identifiers. This is available to all customers.

We recommend setting the business user identifier, where possible, when using either approach.

### Reset generated identifiers on web

If you're not using Identities, there's a risk of attributing sessions to the wrong user when multiple users share the same browser or device.

If this is a problem for you, the JavaScript tracker allows you to [clear all user data](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data). If you call this function when a user logs out, the tracker will generate new user and session identifiers for subsequent events. This feature is only available for the JavaScript tracker.

## Anonymous tracking

The user and session identifiers, along with the IP address, are considered personal identifiable information (PII). Snowplow also allows you to track data without these identifiers: check out the [anonymous tracking](/docs/events/anonymous-tracking/) overview page.

---

# Overview of Snowplow events
> Links to information about Snowplow event types, custom tracking, and out-of-the-box data collection capabilities.
> Source: https://docs.snowplow.io/docs/events/

This section gives an overview of the different kinds of Snowplow events, as well as general tracking concepts.

Read the [custom data](/docs/events/custom-events/) page for information on tracking custom events and entities.

Check out the [out-of-the-box data](/docs/events/ootb-data/) section to learn about built-in tracking APIs.

For an overview of the properties in the atomic events table, see the [atomic event fields reference](/docs/fundamentals/canonical-event/).

---

# Application error events
> Automatically track exceptions and application errors with the application_error event schema on web and mobile.
> Source: https://docs.snowplow.io/docs/events/ootb-data/app-error-events/

Exception (error) tracking captures exceptions within your application.

This table shows the support for exception tracking across the main client-side Snowplow [tracker SDKs](/docs/sources/). The server-side trackers don't include error tracking APIs.

| Tracker                                                                      | Supported | Since version | Auto-tracking | Notes                                                                                                                                                                                                                                |
| ---------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Web](/docs/sources/web-trackers/tracking-events/errors/)                    | ✅         | 3.0.0         | ✅/❌           | Requires error tracking plugin. Track handled exceptions manually.                                                                                                                                                                   |
| [iOS](/docs/sources/mobile-trackers/tracking-events/exception-tracking/)     | ✅         | 1.0.0         | ✅             |                                                                                                                                                                                                                                      |
| [Android](/docs/sources/mobile-trackers/tracking-events/exception-tracking/) | ✅         | 1.0.0         | ✅             |                                                                                                                                                                                                                                      |
| React Native                                                                 | ❌         |               |               | Earlier versions of the React Native tracker supported error tracking for mobile platforms. This feature was removed in version 4.0.0 as it didn't work with JavaScript. Track errors manually using the `application_error` schema. |
| Flutter                                                                      | ❌         |               |               | Use the `application_error` schema for your own custom event.                                                                                                                                                                        |
| Roku                                                                         | ❌         |               |               | Use the `application_error` schema for your own custom event.                                                                                                                                                                        |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)    | ✅         | v3            | ❌             |                                                                                                                                                                                                                                      |

The Errors plugin for the JavaScript trackers provides configuration for automatic tracking of unhandled exceptions, as well as a method for manually tracking handled exceptions.

The mobile trackers will capture unhandled exceptions. These can crash the app, so it's likely that the event will be sent after restart. Note that in some situations, it may not be possible to capture all exception details before the app crashes.

All exception events use the same `application_error` schema, which has the following properties:

### `application_error`

**Type:** Event

Schema for an application error

**Properties:**

| Property                       | Description                                                                                                                                                                                                                                                            |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `programmingLanguage` _string_ | _Required._ Must be one of: `JAVA`, `SCALA`, `KOTLIN`, `GROOVY`, `RUBY`, `GOLANG`, `JAVASCRIPT`, `PHP`, `PYTHON`, `OBJECTIVEC`, `SWIFT`, `C`, `CPLUSPLUS`, `CSHARP`, `ACTIONSCRIPT`, `LUA`, `RUST`, `HASKELL`, `CLOJURE`, `ERLANG`, `ELIXIR`, `CRYSTAL`, `PONY`, `NIM` |
| `message` _string_             | _Required._                                                                                                                                                                                                                                                            |
| `threadName` _string_          | _Optional._                                                                                                                                                                                                                                                            |
| `threadId` _integer_           | _Optional._                                                                                                                                                                                                                                                            |
| `stackTrace` _string_          | _Optional._                                                                                                                                                                                                                                                            |
| `causeStackTrace` _string_     | _Optional._                                                                                                                                                                                                                                                            |
| `lineNumber` _integer_         | _Optional._                                                                                                                                                                                                                                                            |
| `className` _string_           | _Optional._                                                                                                                                                                                                                                                            |
| `exceptionName` _string_       | _Optional._                                                                                                                                                                                                                                                            |
| `isFatal` _boolean_            | _Optional._                                                                                                                                                                                                                                                            |
| `lineColumn` _integer_         | _Optional._                                                                                                                                                                                                                                                            |
| `fileName` _string_            | _Optional._                                                                                                                                                                                                                                                            |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_application_error_1 application_error_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'application_error'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# App and tracker information
> Track application and tracker metadata including version, build, platform, and namespace in atomic fields and context entities.
> Source: https://docs.snowplow.io/docs/events/ootb-data/app-information/

Information about the application and the tracker instance that is sending the events to Snowplow can be useful for data analysis and troubleshooting issues.

You can track this in two ways: setting [atomic event properties](/docs/fundamentals/canonical-event/), or configuring the application entity.

## Application atomic event properties

These [properties](/docs/fundamentals/canonical-event/#application-fields) can be assigned across all our trackers, regardless of the platform.

| Atomic table field | Type | Description                                                                                          | Example values      |
| ------------------ | ---- | ---------------------------------------------------------------------------------------------------- | ------------------- |
| `name_tracker`     | text | The tracker namespace                                                                                | `tracker_1`         |
| `app_id`           | text | Unique identifier for website / application                                                          | `snow-game-android` |
| `platform`         | text | The platform the app runs on                                                                         | `web`, `mob`, `app` |
| `v_tracker`        | text | Identifier for Snowplow tracker. The format follows the convention of `TRACKER_NAME-TRACKER_VERSION` | `js-2.16.2`         |

You can specify the tracker namespace and app ID when creating a new tracker instance.

The tracker platform is set automatically but can be overriden in most of our trackers. The tracker version is also set automatically.

## Application entity

You can configure your web or mobile trackers to automatically include an application entity with all tracked events.

The included data is slightly different depending on the platform, as different schemas are used:

- Web: `version` only
- Mobile: `version` and `build`

This table shows the support for the application entity across the main client-side Snowplow [tracker SDKs](/docs/sources/). The server-side trackers don't include this entity.

| Tracker                                                                                                        | Supported | Since version | Auto-tracking | Notes                                                |
| -------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ---------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#application-version)                                         | ✅         | 4.1.0         | ✅             |                                                      |
| [iOS](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#application-context)     | ✅         | 1.0.0         | ✅             |                                                      |
| [Android](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#application-context) | ✅         | 1.0.0         | ✅             |                                                      |
| [React Native](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/)           | ✅         | 4.0.0         | ✅             | Uses web or mobile schema depending on configuration |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                                     | ✅         | 0.3.0         | ✅             | Available for mobile only                            |
| [Roku](/docs/sources/roku-tracker/adding-data/)                                                                | ✅         | 0.1.0         | ✅             |                                                      |
| Google Tag Manager                                                                                             | ❌         |               |               |                                                      |

The native mobile trackers are able to extract the application version and build number automatically from the app metadata. For web, you need to provide the application version manually in the tracker configuration.

### Entity definitions

Web applications:

### `application`

**Type:** Schema

Schema for an application context which tracks the app version.

**Properties:**

| Property           | Description                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `version` _string_ | _Required._ Version of the application. Can be a semver-like structure (e.g 1.1.0) or a Git commit SHA hash. |

Mobile applications:

### `application`

**Type:** Schema

Schema for an application context which automatically tracks version number and build name when using our mobile SDK's.

**Properties:**

| Property           | Description                                                         |
| ------------------ | ------------------------------------------------------------------- |
| `version` _string_ | _Required._ Version number of the application e.g 1.1.0             |
| `build` _string_   | _Required._ Build name of the application e.g s9f2k2d or 1.1.0 beta |

---

# App performance
> Track web vitals, navigation timing, and custom performance metrics using automatic plugins or manual timing events.
> Source: https://docs.snowplow.io/docs/events/ootb-data/app-performance/

We provide two out-of-the-box solutions for tracking app performance metrics:

- Plugins for automatic performance tracking on web
- Timing events, available in most Snowplow trackers, for manual tracking

## Automatic tracking on web

The [JavaScript tracker](/docs/sources/web-trackers/) provides two configurable plugins that track performance automatically.

Use the web vitals plugin to track key user-centric performance metrics, and the performance navigation timing plugin to capture detailed navigation timing data.

### Web vitals

This [plugin](/docs/sources/web-trackers/tracking-events/web-vitals/) tracks web performance metrics categorized as [Web Vitals](https://web.dev/vitals/).

This table shows which versions of the trackers can use the web vitals plugins:

| Tracker            | Supported | Since version | Auto-tracking | Notes                                                             |
| ------------------ | --------- | ------------- | ------------- | ----------------------------------------------------------------- |
| Web                | ✅         | 3.13.0        | ✅             |                                                                   |
| React Native       | ❌         |               |               | Tracker can't access the required browser APIs                    |
| Google Tag Manager | ❌         |               |               | Not directly supported, but you can load it as an external plugin |

To process raw web vitals event data, use the core web vitals module in the [Snowplow Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/).

### `web_vitals`

**Type:** Event

Schema for a web vitals tracking event. For more information on web vitals you can visit https\://web.dev/vitals/.

**Properties:**

| Property                                   | Description                                                                                                                                                                                                                                                         |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cls` (Cumulative Layout Shift) _number_   | _Optional._ A unitless metric for measuring visual stability because it helps quantify how often users experience unexpected layout shifts. For more information https\://web.dev/cls/.                                                                             |
| `fid` (First Input Delay) _number_         | _Optional._ A metric for measuring load responsiveness because it quantifies the experience users feel when trying to interact with unresponsive pages. Measured in milliseconds. For more information https\://web.dev/fid/.                                       |
| `lcp` (Largest Contentful Paint) _number_  | _Optional._ A metric for measuring perceived load speed because it marks the point in the page load timeline when the page's main content has likely loaded. Measured in milliseconds. For more information https\://web.dev/lcp/.                                  |
| `fcp` (First Contentful Paint) _number_    | _Optional._ A metric for measuring perceived load speed because it marks the first point in the page load timeline where the user can see anything on the screen. Measured in milliseconds. For more information https\://web.dev/fcp/.                             |
| `inp` (Interaction to Next Paint) _number_ | _Optional._ A metric that assesses responsiveness. INP observes the latency of all interactions a user has made with the page, and reports a single value which all (or nearly all) interactions were below that value. For more information https\://web.dev/inp/. |
| `ttfb` (Time To First Byte) _number_       | _Optional._ A DOMHighResTimeStamp referring to the time in milliseconds between the browser requesting a page and when it receives the first byte of information from the server. For more information https\://web.dev/ttfb/.                                      |
| `navigationType` _string_                  | _Optional._ The navigation type recognised from the Navigation Timing API https\://www\.w3.org/TR/navigation-timing-2/. E.g. 'navigate', 'reload', 'back-forward', 'back-forward-cache', 'prerender', 'restore'.                                                    |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_web_vitals_1 web_vitals_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'web_vitals'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Performance navigation timing

This [plugin](/docs/sources/web-trackers/tracking-events/timings/) will add Performance Navigation Timing entities to all tracked events.

This table shows which versions of the trackers can use the performance plugins:

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                                                               |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------------------------------------------- |
| Web                                                                       | ✅         | 3.10.0        | ✅             |                                                                     |
| React Native                                                              | ❌         |               |               | Tracker can't access the required browser APIs to use these plugins |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/) | ✅         | v4            | ✅             | Integrates with the performance navigation timing plugin            |

### `PerformanceNavigationTiming`

**Type:** Schema

Schema for page navigation performance entity, based on the PerformanceNavigationTiming interface (see https\://w3c.github.io/navigation-timing/)

**Properties:**

| Property                              | Description                                                                                                                                                                                                                                                                                                             |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `decodedBodySize` _integer_           | _Optional._ A number that is the size (in octets) received from the fetch (HTTP or cache) of the message body, after removing any applied content encoding.                                                                                                                                                             |
| `encodedBodySize` _integer_           | _Optional._ A number representing the size (in octets) received from the fetch (HTTP or cache), of the payload body, before removing any applied content encodings.                                                                                                                                                     |
| `redirectStart` _number_              | _Optional._ A DOMHighResTimeStamp that represents the start time of the fetch which initiates the redirect in milliseconds.                                                                                                                                                                                             |
| `redirectEnd` _number_                | _Optional._ A DOMHighResTimeStamp immediately after receiving the last byte of the response of the last redirect in milliseconds.                                                                                                                                                                                       |
| `fetchStart` _number_                 | _Optional._ A DOMHighResTimeStamp immediately before the browser starts to fetch the resource in milliseconds.                                                                                                                                                                                                          |
| `domainLookupStart` _number_          | _Optional._ A DOMHighResTimeStamp immediately before the browser starts the domain name lookup for the resource in milliseconds.                                                                                                                                                                                        |
| `domainLookupEnd` _number_            | _Optional._ A DOMHighResTimeStamp representing the time immediately after the browser finishes the domain name lookup for the resource in milliseconds.                                                                                                                                                                 |
| `connectStart` _number_               | _Optional._ A DOMHighResTimeStamp immediately before the browser starts to establish the connection to the server to retrieve the resource in milliseconds.                                                                                                                                                             |
| `secureConnectionStart` _number_      | _Optional._ A DOMHighResTimeStamp immediately before the browser starts the handshake process to secure the current connection in milliseconds.                                                                                                                                                                         |
| `connectEnd` _number_                 | _Optional._ A DOMHighResTimeStamp immediately after the browser finishes establishing the connection to the server to retrieve the resource in milliseconds.                                                                                                                                                            |
| `requestStart` _number_               | _Optional._ A DOMHighResTimeStamp immediately before the browser starts requesting the resource from the server in milliseconds.                                                                                                                                                                                        |
| `responseStart` _number_              | _Optional._ A DOMHighResTimeStamp immediately after the browser receives the first byte of the response from the server in milliseconds.                                                                                                                                                                                |
| `responseEnd` _number_                | _Optional._ A DOMHighResTimeStamp immediately after the browser receives the last byte of the resource or immediately before the transport connection is closed in milliseconds, whichever comes first.                                                                                                                 |
| `unloadEventStart` _number_           | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's unload event handler starts in milliseconds.                                                                                                                                                                           |
| `unloadEventEnd` _number_             | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's unload event handler completes in milliseconds.                                                                                                                                                                        |
| `domInteractive` _number_             | _Optional._ A DOMHighResTimeStamp representing the time immediately before the user agent sets the document's readyState to 'interactive' in milliseconds.                                                                                                                                                              |
| `domContentLoadedEventStart` _number_ | _Optional._ A DOMHighResTimeStamp representing the time immediately before the current document's DOMContentLoaded event handler starts in milliseconds.                                                                                                                                                                |
| `domContentLoadedEventEnd` _number_   | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's DOMContentLoaded event handler completes in milliseconds.                                                                                                                                                              |
| `domComplete` _number_                | _Optional._ A DOMHighResTimeStamp representing the time immediately before the user agent sets the document's readyState to 'complete' in milliseconds.                                                                                                                                                                 |
| `loadEventStart` _number_             | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's load event handler starts in milliseconds.                                                                                                                                                                             |
| `loadEventEnd` _number_               | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's load event handler completes in milliseconds.                                                                                                                                                                          |
| `entryType` _string_                  | _Optional._ The string 'navigation'.                                                                                                                                                                                                                                                                                    |
| `redirectCount` _integer_             | _Optional._ A number representing the number of redirects since the last non-redirect navigation in the current browsing context.                                                                                                                                                                                       |
| `type` _string_                       | _Optional._ A string representing the navigation type. Either 'navigate', 'reload', 'back\_forward' or 'prerender'.                                                                                                                                                                                                     |
| `workerStart` _number_                | _Optional._ Returns a DOMHighResTimeStamp immediately before dispatching the FetchEvent if a Service Worker thread is already running, or immediately before starting the Service Worker thread if it is not already running. If the resource is not intercepted by a Service Worker the property will always return 0. |
| `nextHopProtocol` _string_            | _Optional._ A string representing the network protocol used to fetch the resource, as identified by the ALPN Protocol ID (RFC7301)                                                                                                                                                                                      |
| `transferSize` _integer_              | _Optional._ A number representing the size (in octets) of the fetched resource. The size includes the response header fields plus the response payload body.                                                                                                                                                            |
| `duration` _number_                   | _Optional._ Returns a timestamp that is the difference between the loadEventEnd and startTime properties.                                                                                                                                                                                                               |
| `activationStart` _number_            | _Optional._ If the document is prerendered, activationStart represents the time between when the prerender was started and the document was actually activated.                                                                                                                                                         |
| `deliveryType` _string_               | _Optional._ Expose information about how a resource was delivered e.g. resources which were delivered from the cache.                                                                                                                                                                                                   |
| `serverTiming` _array_                | _Optional._ Array of PerformanceServerTiming entries.                                                                                                                                                                                                                                                                   |

## Manual timing events

Track things like resource load times or other performance measurements using manual timing events.

This table shows the support for timing events across Snowplow [tracker SDKs](/docs/sources/).

| Tracker                                                                                               | Supported | Since version |
| ----------------------------------------------------------------------------------------------------- | --------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/timings/generic/)                                    | ✅         | 3.0.0         |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#creating-a-timing-event)                         | ✅         | 2.0.0         |
| [Android](/docs/sources/mobile-trackers/tracking-events/#creating-a-timing-event)                     | ✅         | 2.0.0         |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-timing-events)            | ✅         | 0.1.0         |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/#track-timing-events-withtiming)              | ✅         | 0.1.0         |
| Roku                                                                                                  | ❌         |               |
| [Java](/docs/sources/java-tracker/tracking-events/#creating-a-timing-event)                           | ✅         | 0.8.0         |
| [Golang](/docs/sources/golang-tracker/tracking-specific-events/#track-timing-events-with-tracktiming) | ✅         | 0.1.0         |
| Python                                                                                                | ❌         |               |
| Ruby                                                                                                  | ❌         |               |
| PHP                                                                                                   | ❌         |               |
| [.NET](/docs/sources/net-tracker/event-tracking/#track-timing-events-withtracktiming)                 | ✅         | 1.0.0         |
| [C++](/docs/sources/c-tracker/tracking-specific-events/#track-timing-events-withtimingevent)          | ✅         | 0.3.0         |
| [Rust](/docs/sources/rust-tracker/tracking-events/#track-timing-events-withtimingevent)               | ✅         | 0.1.0         |
| [Unity](/docs/sources/unity-tracker/event-tracking/#track-timing-events-withtracktiming)              | ✅         | 0.1.0         |
| Scala                                                                                                 | ❌         |               |
| Lua                                                                                                   | ❌         |               |
| Node.js                                                                                               | ❌         |               |
| Google Tag Manager                                                                                    | ❌         |               |

You can still track timing data using trackers without built-in timing event support. Use custom events with the `timing` schema.

### `timing`

**Type:** Event

Schema for a user timing event

**Properties:**

| Property            | Description |
| ------------------- | ----------- |
| `category` _string_ | _Required._ |
| `variable` _string_ | _Required._ |
| `timing` _number_   | _Required._ |
| `label` _string_    | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_timing_1 timing_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'timing'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Campaign and ad tracking
> Track marketing campaigns with UTM parameters and ad performance with impression, click, and conversion events for attribution analysis.
> Source: https://docs.snowplow.io/docs/events/ootb-data/campaigns-and-ads/

Campaign and ad tracking captures how users arrive at your site from marketing campaigns and how they interact with advertisements.

Use campaign and ad tracking to:

- Attribute traffic to specific marketing campaigns
- Measure ad impression and click performance
- Track conversions from ad campaigns
- Analyze the effectiveness of different marketing channels

## Campaign atomic event properties

The [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) can automatically capture UTM parameters from URLs and populate [atomic event properties](/docs/fundamentals/canonical-event/#marketing-fields) for campaign analysis.

## Advertisement events

Ad tracking captures impressions, clicks, and conversions from advertisements displayed on your site or elsewhere.

See also [media tracking](/docs/events/ootb-data/media-events/) for tracking advertising content consumed within video or audio media.

This table shows the support for ad tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                                   | Supported | Since version                       | Auto-tracking | Notes                           |
| ------------------------------------------------------------------------- | --------- | ----------------------------------- | ------------- | ------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/ads/)                    | ✅         | 1.1.0 (directly), 3.0.0 (as plugin) | ❌             | Requires the ad tracking plugin |
| iOS                                                                       | ❌         |                                     |               |                                 |
| Android                                                                   | ❌         |                                     |               |                                 |
| React Native                                                              | ❌         |                                     |               |                                 |
| Flutter                                                                   | ❌         |                                     |               |                                 |
| Roku                                                                      | ❌         |                                     |               |                                 |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3                                  | ❌             |                                 |

### Ad impression event

Track when an ad is displayed to a user.

### `ad_impression`

**Type:** Event

Schema for an ad impression event

**Properties:**

| Property                | Description                                                                      |
| ----------------------- | -------------------------------------------------------------------------------- |
| `impressionId` _string_ | _Optional._ Identifier for the impression instance                               |
| `costModel` _string_    | _Optional._ The cost model: cpa, cpc, or cpm Must be one of: `cpa`, `cpc`, `cpm` |
| `cost` _number_         | _Optional._ The cost of the impression                                           |
| `targetUrl` _string_    | _Optional._ The destination URL                                                  |
| `bannerId` _string_     | _Optional._ Adserver identifier for the ad banner                                |
| `zoneId` _string_       | _Optional._ Adserver identifier for the zone                                     |
| `advertiserId` _string_ | _Optional._ Adserver identifier for the advertiser                               |
| `campaignId` _string_   | _Optional._ Adserver identifier for the campaign                                 |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_ad_impression_1 ad_impression_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_impression'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Ad click event

Track when a user clicks on an ad.

### `ad_click`

**Type:** Event

Schema for an ad click event

**Properties:**

| Property                | Description                                                    |
| ----------------------- | -------------------------------------------------------------- |
| `targetUrl` _string_    | _Required._ The destination URL                                |
| `clickId` _string_      | _Optional._ Identifier for the click instance                  |
| `costModel` _string_    | _Optional._ The cost model Must be one of: `cpa`, `cpc`, `cpm` |
| `cost` _number_         | _Optional._ The cost of the click                              |
| `bannerId` _string_     | _Optional._ Adserver identifier for the ad banner              |
| `zoneId` _string_       | _Optional._ Adserver identifier for the zone                   |
| `impressionId` _string_ | _Optional._ Links click to impression                          |
| `advertiserId` _string_ | _Optional._ Adserver identifier for the advertiser             |
| `campaignId` _string_   | _Optional._ Adserver identifier for the campaign               |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_ad_click_1 ad_click_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_click'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Ad conversion event

Track when a user completes a conversion action from an ad.

### `ad_conversion`

**Type:** Event

Schema for an ad conversion event

**Properties:**

| Property                | Description                                                    |
| ----------------------- | -------------------------------------------------------------- |
| `conversionId` _string_ | _Optional._ Identifier for the conversion instance             |
| `costModel` _string_    | _Optional._ The cost model Must be one of: `cpa`, `cpc`, `cpm` |
| `cost` _number_         | _Optional._ The cost of the conversion                         |
| `category` _string_     | _Optional._ Conversion category                                |
| `action` _string_       | _Optional._ The type of user interaction, e.g. purchase        |
| `property` _string_     | _Optional._ Describes the object of the conversion             |
| `initialValue` _number_ | _Optional._ How much the conversion is initially worth         |
| `advertiserId` _string_ | _Optional._ Adserver identifier for the advertiser             |
| `campaignId` _string_   | _Optional._ Adserver identifier for the campaign               |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_ad_conversion_1 ad_conversion_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_conversion'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Consent events and GDPR
> Track GDPR consent preferences, CMP visibility, and marketing consent with enhanced consent events and legacy consent APIs.
> Source: https://docs.snowplow.io/docs/events/ootb-data/consent-events/

Snowplow provides out-of-the-box [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/) to track user consent preferences and GDPR compliance. Use these to capture consent decisions, track changes to user preferences, and monitor Consent Management Platform (CMP) performance.

You can also configure your tracker not to track certain identifiers, depending on user consent, using [anonymous tracking](/docs/events/anonymous-tracking/).

## Consent API versions

Snowplow provides two versions of consent tracking APIs:

- Enhanced consent APIs: available for web only using the [Enhanced Consent plugin](/docs/sources/web-trackers/tracking-events/consent-gdpr/), these APIs track detailed consent preferences and CMP visibility events
- Basic consent APIs: available for mobile, these APIs track basic consent activity

To process raw events created by the Snowplow Enhanced Consent plugin, use the consent module in the [Snowplow Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/).

> **Note:** The Snowplow dbt packages don't support the basic consent events and entities. You can however build your own models based on the raw events.

## Tracker support

This table shows the support for consent tracking across the main client-side Snowplow [tracker SDKs](/docs/sources/). The server-side trackers don't include consent tracking APIs.

Older versions of the web trackers provided the `browser-plugin-consent` plugin for basic consent tracking. It was deprecated in version 4 in favor of `browser-plugin-enhanced-consent`.

| Tracker                                                                            | Supported | Since version                           | Auto-tracking | Notes                                                                               |
| ---------------------------------------------------------------------------------- | --------- | --------------------------------------- | ------------- | ----------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/consent-gdpr/)                    | ✅         | 3.0.0 - 4.0.0 (basic), 3.8.0 (enhanced) | ✅/❌           | Track enhanced consent events manually; configure automatic addition of GDPR entity |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#creating-a-consent-event)     | ✅         | 1.0.0                                   | ✅/❌           | Track basic consent events manually; configure automatic addition of GDPR entity    |
| [Android](/docs/sources/mobile-trackers/tracking-events/#creating-a-consent-event) | ✅         | 1.0.0                                   | ✅/❌           | Track basic consent events manually; configure automatic addition of GDPR entity    |
| [React Native](/docs/sources/react-native-tracker/tracking-events/)                | ✅         | 1.0.0                                   | ❌             | Basic API\*; no GDPR entity                                                         |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/)                          | ✅         | 0.1.0                                   | ❌             | Basic API                                                                           |
| Roku                                                                               | ❌         |                                         |               | Track custom events using the enhanced schemas                                      |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)          | ✅         | v3                                      | ❌             | Enhanced consent events                                                             |

\*It's also possible to use the JavaScript Enhanced Consent plugin with the React Native tracker.

## Events and entities

This section describes the events and entities used in Snowplow consent tracking.

### Enhanced

The [Enhanced Consent plugin](/docs/sources/web-trackers/tracking-events/consent-gdpr/) includes a number of tracking calls for different user consent actions:

| API                     | To track                                                |
| ----------------------- | ------------------------------------------------------- |
| `trackConsentAllow`     | Acceptance of user consent                              |
| `trackConsentSelected`  | A specific selection of consented scopes                |
| `trackConsentPending`   | The unconfirmed selection about user consent            |
| `trackConsentImplicit`  | The implicit consent on user consent preferences        |
| `trackConsentDeny`      | A denial of user consent                                |
| `trackConsentExpired`   | The expiration of a consent selection                   |
| `trackConsentWithdrawn` | The withdrawal of user consent                          |
| `trackCmpVisible`       | The render time of a consent management platform banner |

With the exception of the CMP visible event, these methods use the same `consent_preferences` event schema. The CMP visible event uses the `cmp_visible` schema.

### `consent_preferences`

**Type:** Event

Schema for consent preferences selection event

**Properties:**

| Property                  | Description                                                                                                                                                                         |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eventType` __            | _Required._ The action for the consent preferences of a user. Must be one of: `deny_all`, `allow_all`, `allow_selected`, `pending`, `implicit_consent`, `withdrawn`, `expired`      |
| `basisForProcessing` __   | _Required._ GDPR lawful basis for data collection & processing. Must be one of: `consent`, `contract`, `legal_obligation`, `vital_interests`, `public_task`, `legitimate_interests` |
| `consentUrl` _string_     | _Required._ URI of the privacy policy related document.                                                                                                                             |
| `consentVersion` _string_ | _Required._ Version of the privacy policy related document.                                                                                                                         |
| `consentScopes` _array_   | _Required._ The scopes allowed after the user finalized his selection of consent preferences. E.g \['analytics', 'functional', 'advertisement'].                                    |
| `domainsApplied` _array_  | _Required._ The domains for which this consent allows these preferences to persist to.                                                                                              |
| `gdprApplies` _boolean_   | _Optional._ Determine if GDPR applies based on the user's geo-location.                                                                                                             |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_consent_preferences_1 consent_preferences_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'consent_preferences'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### `cmp_visible`

**Type:** Event

Tracks the load time of Consent Management Platform (CMP) banners.

**Properties:**

| Property               | Description                                                                 |
| ---------------------- | --------------------------------------------------------------------------- |
| `elapsedTime` _number_ | _Required._ The time taken for the consent popup to be shown to the screen. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_cmp_visible_1 cmp_visible_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'cmp_visible'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Basic

When you track a `consent_granted` or `consent_withdrawn` event, the tracker will automatically create and attach a `consent_document` entity.

The [native mobile](/docs/sources/mobile-trackers/tracking-events/gdpr-tracking/) and [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/) trackers also support automatic addition of the `gdpr` entity to all tracked events. To configure this, provide a `GdprConfiguration` object when setting up a new tracker.

The React Native tracker did include GDPR entity configuration in earlier versions. We [deprecated it in version 4](/docs/sources/react-native-tracker/migration-guides/migrating-from-v2-x-to-v4/).

### `consent_granted`

**Type:** Event

Schema for consent granted

**Properties:**

| Property          | Description |
| ----------------- | ----------- |
| `expiry` _string_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_consent_granted_1 consent_granted_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'consent_granted'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### `consent_withdrawn`

**Type:** Event

Schema for consent withdrawn

**Properties:**

| Property        | Description |
| --------------- | ----------- |
| `all` _boolean_ | _Required._ |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_consent_withdrawn_1 consent_withdrawn_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'consent_withdrawn'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### `consent_document`

**Type:** Schema

Schema for consent document context

**Properties:**

| Property               | Description |
| ---------------------- | ----------- |
| `id` _string_          | _Required._ |
| `version` _string_     | _Required._ |
| `name` _string_        | _Optional._ |
| `description` _string_ | _Optional._ |

### `gdpr`

**Type:** Schema

Schema for a web page context

**Properties:**

| Property                       | Description                                                                                                                                                                 |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `basisForProcessing` _string_  | _Required._ GDPR basis for data collection & processing Must be one of: `consent`, `contract`, `legal_obligation`, `vital_interests`, `public_task`, `legitimate_interests` |
| `documentId` _string_          | _Optional._ ID for document detailing basis for processing                                                                                                                  |
| `documentVersion` _string_     | _Optional._ Version of document detailing basis for processing                                                                                                              |
| `documentDescription` _string_ | _Optional._ Description of document detailing basis for processing                                                                                                          |

---

# Device and browser information
> Track device, browser, and platform information in atomic fields and context entities including YAUAA parsing, Client Hints, and mobile context.
> Source: https://docs.snowplow.io/docs/events/ootb-data/device-and-browser/

Snowplow trackers track information about the device or browser that's sending the events to Snowplow in two ways: setting [atomic event properties](/docs/fundamentals/canonical-event/), or in browser or application entities. You can also choose to configure additional entities through [enrichment](/docs/pipeline/enrichments/).

## Atomic event properties

Track these properties manually or automatically, depending on your tracker.

### Browser

Some trackers can populate the [browser atomic event properties](/docs/fundamentals/canonical-event/#browser-fields): `br_lang`, `br_cookies`, `br_colordepth`, `br_viewheight` and `br_viewwidth`.

Depending on the tracker, you may have to provide the values, as not all trackers capture these fields themselves. The tracker will then add the values automatically to all sent events.

> **Note:** Only the JavaScript tracker can populate the `br_cookies` field.

| Tracker                                                                      | Supported | Since version | Auto-tracking | Notes                                         |
| ---------------------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/browsers/)                  | ✅         | 0.1.0         | ✅             | All values set automatically                  |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.5.0         | ✅             |                                               |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         | ✅             |                                               |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         | ✅             |                                               |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         | ✅             |                                               |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.2.0         | ✅             | Set automatically; language and viewport only |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         | ✅             |                                               |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 1.0.0         | ✅             |                                               |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         | ✅             |                                               |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.10.0        | ✅             |                                               |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.13.0        | ✅             |                                               |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.6.0         | ✅             |                                               |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         | ✅             |                                               |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         | ✅             | Set language only                             |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         | ✅             | Set language and color depth only             |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         | ✅             |                                               |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         | ✅             |                                               |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                   | ✅         | 0.1.0         | ✅             | Set color depth and viewport only             |
| Google Tag Manager                                                           | ❌         |               |               |                                               |

### User agent

No trackers set a [user agent atomic property](/docs/fundamentals/canonical-event/#browser-fields) automatically. See the [YAUAA entity](#yauaa-user-agent-parsing) section below for information about tracking user agents. However, most trackers allow you to manually provide a `useragent`. During enrichment, a provided value will override the one in the HTTP header.

| Tracker                                                                      | Supported | Since version | Auto-tracking |
| ---------------------------------------------------------------------------- | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/browsers/)                  | ✅         | 0.1.0         | ❌             |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.5.0         | ❌             |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         | ❌             |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         | ❌             |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         | ❌             |
| Roku                                                                         | ❌         |               |               |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         | ❌             |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 1.0.0         | ❌             |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         | ❌             |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.8.0         | ❌             |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.6.0         | ❌             |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.3.0         | ❌             |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.4.0         | ❌             |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         | ❌             |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         | ❌             |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         | ❌             |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         | ❌             |
| Lua                                                                          | ❌         |               |               |
| Google Tag Manager                                                           | ❌         |               |               |

### Document

The [document](/docs/fundamentals/canonical-event/#document-fields) atomic properties `doc_charset`, `doc_lang` and `doc_title` are only supported by the [JavaScript tracker](//docs/sources/web-trackers/index.md), which automatically captures these values from the DOM.

### Screen resolution

Some trackers track the [device resolution](/docs/fundamentals/canonical-event/#device-fields) atomic properties `dvce_screenheight` and `dvce_screenwidth` automatically.

| Tracker                                                                      | Supported | Since version | Auto-tracking | Notes                            |
| ---------------------------------------------------------------------------- | --------- | ------------- | ------------- | -------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/browsers/)                  | ✅         | 0.1.0         | ✅             | Autotracking only, no setter API |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.1.0         | ✅             |                                  |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         | ✅             |                                  |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         | ❌             |                                  |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         | ✅/❌           | Autotracking for mobile only     |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.1.0         | ✅             | Autotracking only, no setter API |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         | ❌             |                                  |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 1.0.0         | ❌             |                                  |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         | ❌             |                                  |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.4.0         | ❌             |                                  |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.4.0         | ❌             |                                  |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.1.0         | ❌             |                                  |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         | ❌             |                                  |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         | ❌             |                                  |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         | ❌             |                                  |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         | ❌             |                                  |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         | ❌             |                                  |
| [Lua](/docs/sources/lua-tracker/)                                            | ✅         | 0.1.0         | ❌             |                                  |
| Google Tag Manager                                                           | ❌         |               |               |                                  |

## Entities added by trackers

Configure your trackers to add these entities to tracked events.

### Browser entity

You can configure the web trackers to automatically include a browser entity with all tracked events.

| Tracker                                                                              | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------------------------ | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) | ✅         | 3.9.0         | ✅             |
| iOS                                                                                  | ❌         |               |               |
| Android                                                                              | ❌         |               |               |
| React Native                                                                         | ❌         |               |               |
| Flutter                                                                              | ❌         |               |               |
| Google Tag Manager                                                                   | ❌         |               |               |

The browser entity is only available on web trackers since it captures browser-specific information from the DOM and browser APIs. These APIs aren't available to React Native or Flutter.

### `browser_context`

**Type:** Schema

Schema for browser contexts

**Properties:**

| Property                        | Description                                                                                                                                                   |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `viewport` _string_             | _Required._ Viewport dimensions of the browser. Arrives in the form of WidthxHeight e.g. 1200x900.                                                            |
| `documentSize` _string_         | _Required._ Document dimensions. Arrives in the form of WidthxHeight e.g. 1200x900                                                                            |
| `resolution` _string_           | _Required._ Device native resolution. Arrives in the form of WidthxHeight e.g. 1200x900                                                                       |
| `colorDepth` _integer_          | _Required._ The number of bits allocated to colors for a pixel in the output device, excluding the alpha channel.                                             |
| `devicePixelRatio` _number_     | _Optional._ Ratio of the resolution in physical pixels to the resolution in CSS pixels for the current display device.                                        |
| `cookiesEnabled` _boolean_      | _Required._ Indicates whether cookies are enabled or not. More info and caveats at https\://developer.mozilla.org/en-US/docs/Web/API/Navigator/cookieEnabled. |
| `online` _boolean_              | _Required._ Returns the online status of the browser. Important caveats are described in https\://developer.mozilla.org/en-US/docs/Web/API/Navigator/onLine.  |
| `browserLanguage` _string_      | _Optional._ The preferred language of the user, usually the language of the browser UI. RFC 5646 https\://datatracker.ietf.org/doc/html/rfc5646.              |
| `documentLanguage` _string_     | _Optional._ The language of the HTML document. RFC 5646 https\://datatracker.ietf.org/doc/html/rfc5646.                                                       |
| `webdriver` _boolean_           | _Optional._ Indicates whether the user agent is controlled by automation.                                                                                     |
| `deviceMemory` _integer_        | _Optional._ Approximate amount of device memory in gigabytes.                                                                                                 |
| `hardwareConcurrency` _integer_ | _Optional._ Number of logical processors available to run threads on the user's computer.                                                                     |
| `tabId` _string_                | _Optional._ An identifier for the client browser tab the event is sent from.                                                                                  |

### Mobile entity

The mobile entity gives information about the mobile device platform, including identifiers such as IDFA and IDFV. It's sometimes referred to as the platform entity.

It's enabled by default in all supported trackers. You can configure which of the optional properties to track.

This table shows the support for the mobile entity across the main client-side Snowplow [tracker SDKs](/docs/sources/).

| Tracker                                                                                                               | Supported | Since version | Auto-tracking | Notes                             |
| --------------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------------- |
| Web                                                                                                                   | ❌         |               |               |                                   |
| [iOS](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#platform-mobile-context)        | ✅         | 1.0.0         | ✅             |                                   |
| [Android](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#platform-mobile-context)    | ✅         | 1.0.0         | ✅             |                                   |
| [React Native](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/#platform-context) | ✅         | 1.0.0         | ✅             | Only relevant for mobile tracking |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                                            | ✅         | 0.1.0         | ✅             | Only relevant for mobile tracking |
| Roku                                                                                                                  | ❌         |               |               |                                   |
| Google Tag Manager                                                                                                    | ❌         |               |               |                                   |

### `mobile_context`

**Type:** Schema

Schema for mobile contexts

**Properties:**

| Property                          | Description                                                                                                                                                                                                                 |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `osType` _string_                 | _Required._ Operating system type (e.g., ios, tvos, watchos, osx, android)                                                                                                                                                  |
| `osVersion` _string_              | _Required._ The current version of the operating system                                                                                                                                                                     |
| `deviceManufacturer` _string_     | _Required._ The manufacturer of the product/hardware                                                                                                                                                                        |
| `deviceModel` _string_            | _Required._ The end-user-visible name for the end product                                                                                                                                                                   |
| `carrier` _string_                | _Optional._ The carrier of the SIM inserted in the device                                                                                                                                                                   |
| `networkType` _string_            | _Optional._ Type of network the device is connected to Must be one of: `mobile`, `wifi`, `offline`,``                                                                                                                       |
| `networkTechnology` _string_      | _Optional._ Radio access technology that the device is using                                                                                                                                                                |
| `openIdfa` _string_               | _Optional._ Deprecated tracking identifier for iOS                                                                                                                                                                          |
| `appleIdfa` _string_              | _Optional._ Advertising identifier on iOS                                                                                                                                                                                   |
| `appleIdfv` _string_              | _Optional._ UUID identifier for vendors on iOS                                                                                                                                                                              |
| `androidIdfa` _string_            | _Optional._ Advertising identifier on Android                                                                                                                                                                               |
| `physicalMemory` _integer_        | _Optional._ Total physical system memory in bytes                                                                                                                                                                           |
| `systemAvailableMemory` _integer_ | _Optional._ Available memory on the system in bytes (Android only)                                                                                                                                                          |
| `appAvailableMemory` _integer_    | _Optional._ Amount of memory in bytes available to the current app (iOS only)                                                                                                                                               |
| `batteryLevel` _integer_          | _Optional._ Remaining battery level as an integer percentage of total battery capacity                                                                                                                                      |
| `batteryState` _string_           | _Optional._ Battery state for the device Must be one of: `unplugged`, `charging`, `full`,``                                                                                                                                 |
| `lowPowerMode` _boolean_          | _Optional._ A Boolean indicating whether Low Power Mode is enabled (iOS only)                                                                                                                                               |
| `availableStorage` _integer_      | _Optional._ Bytes of storage remaining                                                                                                                                                                                      |
| `totalStorage` _integer_          | _Optional._ Total size of storage in bytes                                                                                                                                                                                  |
| `isPortrait` _boolean_            | _Optional._ A Boolean indicating whether the device orientation is portrait (either upright or upside down)                                                                                                                 |
| `resolution` _string_             | _Optional._ Screen resolution in pixels. Arrives in the form of WIDTHxHEIGHT (e.g., 1200x900). Doesn't change when device orientation changes                                                                               |
| `scale` _number_                  | _Optional._ Scale factor used to convert logical coordinates to device coordinates of the screen (uses UIScreen.scale on iOS and DisplayMetrics.density on Android)                                                         |
| `language` _string_               | _Optional._ System language currently used on the device (ISO 639)                                                                                                                                                          |
| `appSetId` _string_               | _Optional._ Android vendor ID scoped to the set of apps published under the same Google Play developer account (see https\://developer.android.com/training/articles/app-set-id)                                            |
| `appSetIdScope` _string_          | _Optional._ Scope of the \`appSetId\`. Can be scoped to the app or to a developer account on an app store (all apps from the same developer on the same device will have the same ID) Must be one of: `app`, `developer`,`` |

### Client hints entity

[Client hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Client_hints) are an alternative to user agent strings for capturing browser and device information. You can configure the JavaScript tracker to automatically include a client hints entity with all tracked events.

For other trackers, use the YAUAA enrichment to capture client hints information. See the [YAUAA user agent parsing](#yauaa-user-agent-parsing) section below for more details.

| Tracker                                                         | Supported | Since version | Auto-tracking |
| --------------------------------------------------------------- | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/client-hints/) | ✅         | 3.0.0         | ✅             |
| iOS                                                             | ❌         |               |               |
| Android                                                         | ❌         |               |               |
| React Native                                                    | ❌         |               |               |
| Flutter                                                         | ❌         |               |               |
| Roku                                                            | ❌         |               |               |
| Google Tag Manager                                              | ❌         |               |               |

### `http_client_hints`

**Type:** Schema

Schema for HTTP Client Hints context

**Properties:**

| Property                   | Description                                           |
| -------------------------- | ----------------------------------------------------- |
| `isMobile` _boolean_       | _Optional._ Whether the browser is on a mobile device |
| `brands` _array_           | _Optional._ List of browser brands and versions       |
| `architecture` _string_    | _Optional._ CPU architecture (high entropy)           |
| `model` _string_           | _Optional._ Device model (high entropy)               |
| `platform` _string_        | _Optional._ Operating system name (high entropy)      |
| `platformVersion` _string_ | _Optional._ Operating system version (high entropy)   |
| `uaFullVersion` _string_   | _Optional._ Full browser version (high entropy)       |

## Entities added during enrichment

You can configure your pipeline to add these entities to tracked web events.

### YAUAA user agent parsing

The [YAUAA (Yet Another User Agent Analyzer) enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enables parsing the user agent string tracked in web events. It extracts information about the user's device and browser.

This enrichment can also use client hints headers.

### `yauaa_context`

**Type:** Schema

Schema for a context generated by the YAUAA enrichment after parsing the user agent

**Properties:**

| Property                                   | Description                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `deviceClass` __                           | _Required._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Desktop`, `Anonymized`, `Unknown`, `UNKNOWN`, `Mobile`, `Tablet`, `Phone`, `Watch`, `Virtual Reality`, `eReader`, `Set-top box`, `TV`, `Game Console`, `Home Appliance`, `Handheld Game Console`, `Voice`, `Car`, `Robot`, `Robot Mobile`, `Spy`, `Hacker`, `Augmented Reality`, `Robot Imitator` |
| `deviceName` _string_                      | _Optional._ Example: Google Nexus 6                                                                                                                                                                                                                                                                                                                                            |
| `deviceBrand` _string_                     | _Optional._ Example: Google                                                                                                                                                                                                                                                                                                                                                    |
| `deviceCpu` _string_                       | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `deviceCpuBits` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `deviceFirmwareVersion` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `deviceVersion` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemClass` _string_            | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Desktop`, `Mobile`, `Cloud`, `Embedded`, `Game Console`, `Hacker`, `Anonymized`, `Unknown`                                                                                                                                                                                                        |
| `operatingSystemName` _string_             | _Optional._ Examples: Linux, Android.                                                                                                                                                                                                                                                                                                                                          |
| `operatingSystemVersion` _string_          | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemNameVersion` _string_      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemVersionBuild` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineClass` _string_               | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Browser`, `Mobile App`, `Hacker`, `Robot`, `Unknown`, `Special`, `Cloud`, `eReader`                                                                                                                                                                                                               |
| `layoutEngineName` _string_                | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineVersion` _string_             | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineVersionMajor` _string_        | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineNameVersion` _string_         | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineNameVersionMajor` _string_    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineBuild` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentClass` _string_                      | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Browser`, `Browser Webview`, `Mobile App`, `Robot`, `Robot Mobile`, `Cloud Application`, `Email Client`, `Voice`, `Special`, `Testclient`, `Hacker`, `Unknown`, `Desktop App`, `eReader`                                                                                                          |
| `agentName` _string_                       | _Optional._ Example: Chrome.                                                                                                                                                                                                                                                                                                                                                   |
| `agentVersion` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentVersionMajor` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentNameVersion` _string_                | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentNameVersionMajor` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentBuild` _string_                      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentLanguage` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentLanguageCode` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentInformationEmail` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentInformationUrl` _string_             | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentSecurity` _string_                   | _Optional._ Must be one of: `Weak security`, `Strong security`, `Unknown`, `Hacker`, `No security`                                                                                                                                                                                                                                                                             |
| `agentUuid` _string_                       | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppName` _string_                  | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppVersion` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppVersionMajor` _string_          | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppNameVersionMajor` _string_      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookCarrier` _string_                 | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceClass` _string_             | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceName` _string_              | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceVersion` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookFBOP` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookFBSS` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookOperatingSystemName` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookOperatingSystemVersion` _string_  | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `anonymized` _string_                      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `hackerAttackVector` _string_              | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `hackerToolkit` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `koboAffiliate` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `koboPlatformId` _string_                  | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityVersion` _string_          | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityVersionMajor` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityNameVersion` _string_      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityNameVersionMajor` _string_ | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `carrier` _string_                         | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `gSAInstallationID` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `networkType` _string_                     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemNameVersionMajor` _string_ | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemVersionMajor` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |

### Spiders and robots

The [IAB Spiders and Robots enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) uses the IAB/ABC International Spiders and Bots List to determine whether an event was produced by a user or a robot or spider based on its IP address and user agent.

### `spiders_and_robots`

**Type:** Schema

Schema for a context generated by the IAB Spiders & Robots enrichment

**Properties:**

| Property                  | Description                                                                                                                                                                                                                  |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spiderOrRobot` _boolean_ | _Required._ true if the IP address or user agent checked against the list is a spider or robot, false otherwise                                                                                                              |
| `category` __             | _Required._ Category based on activity if the IP/UA is a spider or robot, BROWSER otherwise Must be one of: `SPIDER_OR_ROBOT`, `ACTIVE_SPIDER_OR_ROBOT`, `INACTIVE_SPIDER_OR_ROBOT`, `BROWSER`                               |
| `reason` __               | _Required._ Type of failed check if the IP/UA is a spider or robot, PASSED\_ALL otherwise Must be one of: `FAILED_IP_EXCLUDE`, `FAILED_UA_INCLUDE`, `FAILED_UA_EXCLUDE`, `PASSED_ALL`                                        |
| `primaryImpact` __        | _Required._ Whether the spider or robot would affect page impression measurement, ad impression measurement, both or none Must be one of: `PAGE_IMPRESSIONS`, `AD_IMPRESSIONS`, `PAGE_AND_AD_IMPRESSIONS`, `UNKNOWN`, `NONE` |

---

# Ecommerce events and entities
> Track ecommerce transactions, carts, checkouts, products, and promotions with Snowplow ecommerce events and context entities.
> Source: https://docs.snowplow.io/docs/events/ootb-data/ecommerce-events/

The Snowplow ecommerce tracking APIs enable you to track user behavior and interactions across your ecommerce store. Track product views, cart interactions, checkout steps, transactions, promotions, and more.

Use ecommerce tracking to answer questions such as:

- Which products do users view but not add to cart?
- Where do users drop off in the checkout process?
- How do promotions affect purchase behavior?
- What is the average order value by user segment?
- Which products are frequently purchased together?

## Tracker support

This table shows the support for ecommerce tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers can send ecommerce events as custom [self-describing events](/docs/fundamentals/events/#self-describing-events).

| Tracker                                                                        | Supported | Since version | Auto-tracking | Notes                                                |
| ------------------------------------------------------------------------------ | --------- | ------------- | ------------- | ---------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/ecommerce/)                   | ✅         | 3.7.0         | ❌             | Requires ecommerce plugin                            |
| [iOS](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/)       | ✅         | 5.4.0         | ❌             |                                                      |
| [Android](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/)   | ✅         | 5.4.0         | ❌             |                                                      |
| [React Native](/docs/sources/react-native-tracker/)                            | ✅         | 4.2.0         | ❌             | Requires ecommerce plugin\*                          |
| Flutter                                                                        | ❌         |               |               |                                                      |
| Roku                                                                           | ❌         |               |               | Use the ecommerce schemas for your own custom events |
| [Google Tag Manager](/docs/sources/google-tag-manager/ecommerce-tag-template/) | ✅         | v3            | ❌             |                                                      |

\*You can use the [JavaScript ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/) and APIs for React Native ecommerce tracking.

The Snowplow ecommerce tracking APIs are supported by the [Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) dbt data model. It's a fully incremental model that transforms raw ecommerce event data into a set of derived tables based around carts, checkouts, products, and transactions.

We recommend using the [Ecommerce base tracking plan template](/docs/event-studio/tracking-plans/templates/#ecommerce-web-and-mobile) for ecommerce tracking.

## Available events

The ecommerce APIs include a number of tracking functions for different ecommerce actions. The exact API varies between trackers, but the available events are consistent across all supported trackers.

| Event behavior     | Used for                                                                                                                                       |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Product view       | Tracking a visit to a product detail screen. Also known as product detail view.                                                                |
| Add to cart        | Track an addition to cart.                                                                                                                     |
| Remove from cart   | Track a removal from cart.                                                                                                                     |
| Product list view  | Track an impression of a product list. The list could be a search results page, recommended products, upsells etc.                             |
| Product list click | Track the click/selection of a product from a product list.                                                                                    |
| Promotion view     | Track an impression for an internal promotion banner or slider or any other type of content that showcases internal products/categories.       |
| Promotion click    | Track the click/selection of an internal promotion.                                                                                            |
| Checkout step      | Track a checkout step completion in the checkout process together with common step attributes for user choices throughout the checkout funnel. |
| Transaction        | Track a transaction/purchase completion.                                                                                                       |
| Transaction error  | Track a failed transaction.                                                                                                                    |
| Refund             | Track a transaction partial or complete refund.                                                                                                |

All these events use the same underlying schema. They're distinguished by their `type` property.

### `snowplow_ecommerce_action`

**Type:** Event

Schema for an Ecommerce action

**Properties:**

| Property        | Description                                                                                                                                                                                                               |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` __       | _Required._ Standard ecommerce actions. Must be one of: `add_to_cart`, `remove_from_cart`, `product_view`, `list_click`, `list_view`, `promo_click`, `promo_view`, `checkout_step`, `transaction`, `refund`, `trns_error` |
| `name` _string_ | _Optional._ The name of the list presented to the user E.g. product list, search results, shop the look, frequently bought with.                                                                                          |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1 snowplow_ecommerce_action_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'snowplow_ecommerce_action'

and events.event_vendor = 'com.snowplowanalytics.snowplow.ecommerce'
```

## Automatically included entities

Every ecommerce event includes entities that describe the user interaction. These entities are attached automatically by the tracker. You don't need to track them yourself.

### Cart

The cart entity captures the current state of the shopping cart, including its total value and currency.

### `cart`

**Type:** Schema

Schema for a cart entity in Ecommerce

**Properties:**

| Property               | Description                                                     |
| ---------------------- | --------------------------------------------------------------- |
| `cart_id` _string_     | _Optional._ The unique ID representing this cart.               |
| `total_value` _number_ | _Required._ The total value of the cart after this interaction. |
| `currency` _string_    | _Required._ The currency used for this cart (ISO 4217).         |

### Checkout step

The checkout step entity tracks information about a specific step in the checkout process, including delivery details, payment method, and user preferences.

### `checkout_step`

**Type:** Schema

Schema for a checkout step entity in Ecommerce

**Properties:**

| Property                         | Description                                                                                                                         |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `step` _integer_                 | _Required._ Checkout step index.                                                                                                    |
| `shipping_postcode` _string_     | _Optional._ Shipping address postcode.                                                                                              |
| `billing_postcode` _string_      | _Optional._ Billing address postcode.                                                                                               |
| `shipping_full_address` _string_ | _Optional._ Full shipping address.                                                                                                  |
| `billing_full_address` _string_  | _Optional._ Full billing address.                                                                                                   |
| `delivery_provider` _string_     | _Optional._ Can be used to discern delivery providers DHL, PostNL etc.                                                              |
| `delivery_method` _string_       | _Optional._ Can be used to discern delivery methods selected E.g. store pickup, standard delivery, express delivery, international. |
| `coupon_code` _string_           | _Optional._ Coupon applied at checkout.                                                                                             |
| `account_type` _string_          | _Optional._ Type of account used on checkout. E.g. existing user, guest.                                                            |
| `payment_method` _string_        | _Optional._ Any kind of payment method the user selected to proceed E.g. card, PayPal, Alipay etc.                                  |
| `proof_of_payment` _string_      | _Optional._ Invoice or receipt.                                                                                                     |
| `marketing_opt_in` _boolean_     | _Optional._ If opted in to marketing campaigns.                                                                                     |

### Product

The product entity contains detailed information about a product, including its ID, category, price, and other attributes such as brand, variant, and inventory status.

### `product`

**Type:** Schema

Schema for a product entity in Ecommerce

**Properties:**

| Property                    | Description                                                                                                                       |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `id` _string_               | _Required._ The SKU or product ID.                                                                                                |
| `name` _string_             | _Optional._ The name or title of the product.                                                                                     |
| `category` _string_         | _Required._ The category the product belongs to. Use a consistent separator to express multiple levels. E.g. Woman/Shoes/Sneakers |
| `price` _number_            | _Required._ The price of the product at the current time.                                                                         |
| `list_price` _number_       | _Optional._ The list or recommended retail price of a product.                                                                    |
| `quantity` _integer_        | _Optional._ The quantity of the product taking part in the ecommerce action.                                                      |
| `size` _string_             | _Optional._ The size of the product.                                                                                              |
| `variant` _string_          | _Optional._ The variant of the product.                                                                                           |
| `brand` _string_            | _Optional._ The brand of the product.                                                                                             |
| `inventory_status` _string_ | _Optional._ The inventory status of the product E.g. in stock, out of stock, preorder, backorder.                                 |
| `position` _integer_        | _Optional._ The position the product was presented in a list of products E.g. search results, product list page.                  |
| `currency` _string_         | _Required._ The currency in which the product is being priced (ISO 4217).                                                         |
| `creative_id` _string_      | _Optional._ Identifier/Name/Url for the creative presented on a list or product view.                                             |

### Promotion

The promotion entity tracks internal promotional content such as banners, sliders, or featured product showcases.

### `promotion`

**Type:** Schema

Schema for a promotion entity in Ecommerce

**Properties:**

| Property               | Description                                                                                                                     |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `id` _string_          | _Required._ The ID of the promotion.                                                                                            |
| `name` _string_        | _Optional._ The name of the promotion.                                                                                          |
| `product_ids` _array_  | _Optional._ Array of SKUs or product IDs showcased in the promotion.                                                            |
| `position` _integer_   | _Optional._ The position the promotion was presented in a list of promotions E.g. banner, slider.                               |
| `creative_id` _string_ | _Optional._ Identifier/Name/Url for the creative presented on the promotion.                                                    |
| `type` _string_        | _Optional._ Type of the promotion delivery mechanism. E.g. popup, banner, intra-content                                         |
| `slot` _string_        | _Optional._ The website slot in which the promotional content was added to. E.g. Identifier for slot sidebar-1, intra-content-2 |

### Refund

The refund entity captures information about a transaction refund, including the amount refunded and the reason for the refund.

### `refund`

**Type:** Schema

Schema for a refund in Ecommerce

**Properties:**

| Property                  | Description                                                               |
| ------------------------- | ------------------------------------------------------------------------- |
| `transaction_id` _string_ | _Required._ The ID of the transaction.                                    |
| `currency` _string_       | _Required._ The currency in which the product is being priced (ISO 4217). |
| `refund_amount` _number_  | _Required._ The monetary amount refunded.                                 |
| `refund_reason` _string_  | _Optional._ Reason for refunding the whole or part of the transaction.    |

### Transaction

The transaction entity contains details about a completed purchase, including revenue, payment method, shipping costs, and applied discounts.

### `transaction`

**Type:** Schema

Schema for a transaction entity in Ecommerce

**Properties:**

| Property                   | Description                                                   |
| -------------------------- | ------------------------------------------------------------- |
| `transaction_id` _string_  | _Required._ The ID of the transaction.                        |
| `revenue` _number_         | _Required._ The revenue of the transaction.                   |
| `currency` _string_        | _Required._ The currency used for the transaction (ISO 4217). |
| `payment_method` _string_  | _Required._ The payment method used for the transaction.      |
| `total_quantity` _integer_ | _Required._ Total quantity of items in the transaction.       |
| `tax` _number_             | _Optional._ Total amount of tax on the transaction.           |
| `shipping` _number_        | _Optional._ Total cost of shipping on the transaction.        |
| `discount_code` _string_   | _Optional._ Discount code used.                               |
| `discount_amount` _number_ | _Optional._ Discount amount taken off.                        |
| `credit_order` _boolean_   | _Optional._ Whether the transaction is a credit order or not. |

### Transaction error

The transaction error entity captures information about failed transactions, including error codes, descriptions, and error types.

### `transaction_error`

**Type:** Schema

Schema for a transaction error or rejection entity in ecommerce.

**Properties:**

| Property                     | Description                                                                                                                                                                                                                                                               |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `error_code` _string_        | _Optional._ Error-identifying code for the transaction issue. E.g. E522                                                                                                                                                                                                   |
| `error_shortcode` _string_   | _Optional._ Shortcode for the error occurred in the transaction. E.g. declined\_by\_stock\_api, declined\_by\_payment\_method, card\_declined, pm\_card\_radarBlock                                                                                                       |
| `error_description` _string_ | _Optional._ Longer description for the error occurred in the transaction.                                                                                                                                                                                                 |
| `error_type` _string_        | _Optional._ Hard error types mean the customer must provide another form of payment e.g. an expired card. Soft errors can be the result of temporary issues where retrying might be successful e.g. processor declined the transaction. Must be one of: `hard`, `soft`,`` |
| `resolution` _string_        | _Optional._ The resolution selected for the error scenario. E.g. retry\_allowed, user\_blacklisted, block\_gateway, contact\_user, default                                                                                                                                |

## Global ecommerce entities

You can configure the ecommerce `user` and `page` entities to automatically attach to **all** Snowplow events tracked by the tracker, not just the ecommerce events. This is useful if you want to have ecommerce user and page information available across your entire dataset.

### Page

The page entity describes the type of ecommerce page being viewed, such as homepage, product page, or checkout page.

### `page`

**Type:** Schema

Schema for a page entity in Ecommerce

**Properties:**

| Property            | Description                                                                                   |
| ------------------- | --------------------------------------------------------------------------------------------- |
| `type` _string_     | _Required._ The type of the page that was visited E.g. homepage, product page, checkout page. |
| `language` _string_ | _Optional._ The language that the web page is based in.                                       |
| `locale` _string_   | _Optional._ The locale version of the site that is running.                                   |

### User

The user entity provides information about the user making the purchase, including their ID, email address, and whether they're a guest.

### `user`

**Type:** Schema

Schema for an user entity in Ecommerce

**Properties:**

| Property             | Description                                     |
| -------------------- | ----------------------------------------------- |
| `id` _string_        | _Required._ The user ID.                        |
| `is_guest` _boolean_ | _Optional._ Whether or not the user is a guest. |
| `email` _string_     | _Optional._ The user's email address.           |

## Legacy ecommerce events

Some Snowplow trackers also provide tracking calls for older ecommerce events. We strongly recommend using the new ecommerce tracking API instead.

The legacy transaction events populate the [transaction atomic event properties](/docs/fundamentals/canonical-event/#legacy-ecommerce-fields): `tr_` fields for transaction events and `ti_` fields for transaction item events.

Transaction event properties:

| **Parameter** | **Table Column** | **Type** | **Description**                                     | **Example values** |
| ------------- | ---------------- | -------- | --------------------------------------------------- | ------------------ |
| `tr_id`       | `tr_orderid`     | text     | Order ID                                            | `12345`            |
| `tr_af`       | `tr_affiliation` | text     | Transaction affiliation (e.g. channel)              | `Web`              |
| `tr_tt`       | `tr_total`       | decimal  | Transaction total value                             | `9.99`             |
| `tr_tx`       | `tr_tax`         | decimal  | Transaction tax value (i.e. amount of VAT included) | `1.98`             |
| `tr_sh`       | `tr_shipping`    | decimal  | Delivery cost charged                               | `3.00`             |
| `tr_ci`       | `tr_city`        | text     | Delivery address: city                              | `London`           |
| `tr_st`       | `tr_state`       | text     | Delivery address: state                             | `Denver`           |
| `tr_co`       | `tr_country`     | text     | Delivery address: country                           | `United Kingdom`   |
| `tr_cu`       | `tr_currency`    | text     | Transaction Currency                                | `GBP`              |

Transaction item event properties:

| **Parameter** | **Table Column** | **Type** | **Description** | **Example values** |
| ------------- | ---------------- | -------- | --------------- | ------------------ |
| `ti_id`       | `ti_orderid`     | text     | Order ID        | `12345`            |
| `ti_sk`       | `ti_sku`         | text     | Item SKU        | \`pbz0025'         |
| `ti_nm`       | `ti_name`        | text     | Item name       | `black-tarot`      |
| `ti_ca`       | `ti_category`    | text     | Item category   | `tarot`            |
| `ti_pr`       | `ti_price`       | decimal  | Item price      | `7.99`             |
| `ti_qu`       | `ti_quantity`    | integer  | Item quantity   | `2`                |
| `ti_cu`       | `ti_currency`    | text     | Currency        | `USD`              |

---

# Geolocation and timezone tracking
> Track user location using IP-based geolocation enrichment or device GPS coordinates with the geolocation context entity.
> Source: https://docs.snowplow.io/docs/events/ootb-data/geolocation/

Snowplow provides several ways to capture location and timezone information:

- IP-based geolocation: the IP Lookup enrichment adds location data to the `geo_` atomic event properties based on the user's IP address.
- Device geolocation: client-side trackers can access GPS coordinates from the device and attach them as an entity. This requires user permission but provides more accurate location data.
- Timezone: most trackers can capture the device timezone and add it to the `os_timezone` atomic event property.

## Location atomic event properties

The [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) populates the `geo_` [atomic event properties](/docs/fundamentals/canonical-event/#location-fields), based on the IP address of the HTTP request to the event Collector.

## Timezone atomic event property

Some Snowplow trackers can populate the `os_timezone` [atomic event field](/docs/fundamentals/canonical-event/#time-and-date-fields) based on the device timezone.

Depending on the tracker, you may need to configure a plugin, set configuration on tracker initialization, or use the tracker Subject class. This table shows the support for timezone tracking across the main [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                      | Supported | Since version |
| ---------------------------------------------------------------------------- | --------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)      | ✅         | 3.0.0         |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.1.0         |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.1.0         |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 0.1.0         |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.4.0         |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.4.0         |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.1.0         |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         |
| Lua                                                                          | ❌         |               |
| Google Tag Manager                                                           | ❌         |               |

## Geolocation entity

Some Snowplow trackers can access device geolocation information, e.g. GPS coordinates, and attach it to events as an entity. This requires permission from the user, but is often more accurate than the IP-based geolocation.

This table shows the support for geolocation tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include geolocation tracking.

| Tracker                                                                    | Supported | Since version | Auto-tracking | Notes                           |
| -------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)    | ✅         | 3.0.0         | ✅             | Requires timezone plugin        |
| [iOS](/docs/sources/mobile-trackers/installation-and-set-up/)              | ✅         | 0.6.0         | ✅             | Set in tracker configuration    |
| [Android](/docs/sources/mobile-trackers/installation-and-set-up/)          | ✅         | 0.6.0         | ✅             | Set in tracker configuration    |
| React Native                                                               | ❌         |               |               |                                 |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/) | ✅         | 0.1.0         | ✅             | Set in tracker configuration    |
| Roku                                                                       | ❌         |               |               |                                 |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)  | ✅         | v3            | ❌             | Integrates with timezone plugin |

The React Native tracker did include geolocation entity configuration in earlier versions. We [deprecated it in version 4](/docs/sources/react-native-tracker/migration-guides/migrating-from-v2-x-to-v4/).

### `geolocation_context`

**Type:** Schema

Schema for client geolocation contexts

**Properties:**

| Property                             | Description |
| ------------------------------------ | ----------- |
| `latitude` _number_                  | _Required._ |
| `longitude` _number_                 | _Required._ |
| `latitudeLongitudeAccuracy` _number_ | _Optional._ |
| `altitude` _number_                  | _Optional._ |
| `altitudeAccuracy` _number_          | _Optional._ |
| `bearing` _number_                   | _Optional._ |
| `speed` _number_                     | _Optional._ |
| `timestamp` _integer_                | _Optional._ |

---

# Out-of-the-box data types
> Overview of Snowplow's built-in event types and entities for web and mobile tracking including ecommerce, media, and performance data.
> Source: https://docs.snowplow.io/docs/events/ootb-data/

Snowplow has provided these out-of-the-box [event](/docs/fundamentals/events/) and [entity](/docs/fundamentals/entities/) types so you can get started quickly with tracking user data.

This table shows the available out-of-the-box data grouped by category.

| Category                                                                                   | Description                                                                         | Events | Entities | [Atomic fields](/docs/fundamentals/canonical-event/) |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | ------ | -------- | ---------------------------------------------------- |
| [Application errors](/docs/events/ootb-data/app-error-events/)                             | Exception messages, stack traces, and error classification                          | ✅      |          |                                                      |
| [Application and tracker information](/docs/events/ootb-data/app-information/)             | App version, build number, platform, and tracker namespace                          |        | ✅        | ✅                                                    |
| [Application performance](/docs/events/ootb-data/app-performance/)                         | Web vitals, navigation timing, and performance metrics                              | ✅      | ✅        |                                                      |
| [Buttons](/docs/events/ootb-data/page-elements/)                                           | Button click events with label, ID, and element classes                             | ✅      |          |                                                      |
| [Campaigns and ads](/docs/events/ootb-data/campaigns-and-ads/)                             | UTM parameters, ad impressions, clicks, and conversions                             | ✅      |          | ✅                                                    |
| [Consent](/docs/events/ootb-data/consent-events/)                                          | GDPR consent and CMP visibility tracking                                            | ✅      | ✅        |                                                      |
| [Device and browser information](/docs/events/ootb-data/device-and-browser/)               | Track data such as language, viewport size, screen resolution, or client hints      |        | ✅        | ✅                                                    |
| [Ecommerce](/docs/events/ootb-data/ecommerce-events/)                                      | Complete ecommerce tracking including product views, cart actions, and transactions | ✅      | ✅        | ✅                                                    |
| [Forms](/docs/events/ootb-data/page-elements/)                                             | Form focus, field changes, and submission with field values                         | ✅      |          |                                                      |
| [Geolocation](/docs/events/ootb-data/geolocation/)                                         | IP-based location (city, region, country) and GPS coordinates                       |        | ✅        | ✅                                                    |
| [Referrers, links, and cross-navigation](/docs/events/ootb-data/links-and-referrers/)      | Page referrer URLs, link clicks, and mobile deep links                              | ✅      | ✅        | ✅                                                    |
| [Media](/docs/events/ootb-data/media-events/)                                              | Complete video or audio tracking, including play, pause, and ad breaks              | ✅      | ✅        |                                                      |
| [Mobile application lifecycle](/docs/events/ootb-data/mobile-lifecycle-events/)            | App install, foreground and background                                              | ✅      | ✅        |                                                      |
| [Page and screen engagement](/docs/events/ootb-data/page-activity-tracking/)               | Page pings, scroll depth, and time engaged on page or screen                        | ✅      | ✅        | ✅                                                    |
| [Page and screen views](/docs/events/ootb-data/page-and-screen-view-events/)               | Track page and screen views                                                         | ✅      | ✅        | ✅                                                    |
| [Page elements](/docs/events/ootb-data/page-elements/)                                     | Element visibility, scroll depth into elements, and lifecycle                       | ✅      | ✅        |                                                      |
| [Site search](/docs/events/ootb-data/site-search/)                                         | Search terms, filters applied, and result counts                                    | ✅      |          |                                                      |
| [Social media interactions](/docs/events/ootb-data/social-media/)                          | Likes, shares, and social widget interactions by network                            | ✅      |          |                                                      |
| [Third-party sources](/docs/events/ootb-data/third-party-sources/)                         | Optimizely experiments, GA cookies, Kantar Focal Meter, and Privacy Sandbox topics  |        | ✅        |                                                      |
| [Timezone](/docs/events/ootb-data/geolocation/)                                            | Track device timezone                                                               |        |          | ✅                                                    |
| [User and session identification](/docs/events/ootb-data/user-and-session-identification/) | Track identifiers such as user ID, domain user ID, session ID, and session index    |        | ✅        | ✅                                                    |
| [VisionOS and SwiftUI](/docs/events/ootb-data/visionos-swiftui/)                           | Window group and immersive space tracking for Apple                                 | ✅      | ✅        |                                                      |

---

# Referrers, links, and cross-navigation events
> Track page referrers, deep links in mobile apps, and cross-navigation links.
> Source: https://docs.snowplow.io/docs/events/ootb-data/links-and-referrers/

Understanding how users arrive at your pages and navigate between them is essential for analyzing traffic sources, marketing effectiveness, and user journeys. Snowplow captures referrer information, deep link data, and link click events to give you a complete picture of cross-page and cross-app navigation.

## Referrer atomic event property

Some Snowplow trackers can populate the `page_referrer` [atomic event property](/docs/fundamentals/canonical-event/#page-fields). This field captures the URL of the page that referred the user to the current page.

This table shows the support for page referrer tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include referrer tracking.

| Tracker                                                                                                | Supported | Since version                                      | Auto-tracking | Notes                              |
| ------------------------------------------------------------------------------------------------------ | --------- | -------------------------------------------------- | ------------- | ---------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/page-views/)                                          | ✅         | 0.1.0                                              | ✅             | Automatic for all page view events |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                              | ✅         | 3.0.0 (deep link events), 4.1.0 (deep link entity) | ❌             | Based on Deep Link events          |
| [Android](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                          | ✅         | 3.0.0 (deep link events), 4.1.0 (deep link entity) | ❌             | Based on Deep Link events          |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-deep-link-received-events) | ✅         | 1.1.0                                              | ❌             | Based on Deep Link events          |
| Flutter                                                                                                | ❌         |                                                    |               |                                    |
| Roku                                                                                                   | ❌         |                                                    |               |                                    |
| Google Tag Manager                                                                                     | ✅         | v3                                                 | ✅             | Automatic for all page view events |

The web trackers automatically populate the `page_referrer` [atomic event property](/docs/fundamentals/canonical-event/#page-fields) in all page view events.

### Deep links for mobile

Deep links are URLs or hyperlinks that take users directly to a particular location within a mobile app. They're received by the mobile operating system and passed to the related app.

Most events tracked on mobile unsurprisingly don't include webpage atomic properties. However, the mobile trackers can capture both the URL of the deep link and the referrer URL (if available).

You'll need to manually track any received deep links, using the `deep_link_received` event.

The `deep_link_received` event, as well as the first subsequent screen view event, include the `page_url` and `page_referrer` atomic event properties.

### `deep_link_received`

**Type:** Event

Represents a deep-link received in the app.

**Properties:**

| Property            | Description                                        |
| ------------------- | -------------------------------------------------- |
| `url` _string_      | _Required._ URL in the received deep-link          |
| `referrer` _string_ | _Optional._ Referrer URL, source of this deep-link |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_mobile_deep_link_received_1 deep_link_received_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'deep_link_received'

and events.event_vendor = 'com.snowplowanalytics.mobile'
```

The tracker will automatically add this `deep_link` entity to the next screen view event after tracking the `deep_link_received` event.

### `deep_link`

**Type:** Schema

Entity that indicates a deep-link has been received and processed.

**Properties:**

| Property            | Description                                        |
| ------------------- | -------------------------------------------------- |
| `url` _string_      | _Required._ URL in the received deep-link          |
| `referrer` _string_ | _Optional._ Referrer URL, source of this deep-link |

## Link clicks

The web trackers can automatically track link click events, capturing details about the clicked link such as its URL, element ID, classes, target, and content.

This table shows the support for link click tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include link click tracking.

| Tracker                                                                   | Supported | Since version                                 | Auto-tracking | Notes                             |
| ------------------------------------------------------------------------- | --------- | --------------------------------------------- | ------------- | --------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/link-click/)             | ✅         | 0.7.0 (limited functionality), 3.0.0 (plugin) | ✅             | Requires link click plugin        |
| iOS                                                                       | ❌         |                                               |               |                                   |
| Android                                                                   | ❌         |                                               |               |                                   |
| React Native                                                              | ❌         |                                               |               |                                   |
| Flutter                                                                   | ❌         |                                               |               |                                   |
| Roku                                                                      | ❌         |                                               |               |                                   |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3                                            | ✅             | Integrates with link click plugin |

To track link clicks on web using React Native or Flutter, you can implement custom event tracking when users interact with links in your app using the `link_click` event schema.

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes link clicks.

### `link_click`

**Type:** Event

Schema for a link click event

**Properties:**

| Property                  | Description |
| ------------------------- | ----------- |
| `elementId` _string_      | _Optional._ |
| `elementClasses` _array_  | _Optional._ |
| `elementTarget` _string_  | _Optional._ |
| `targetUrl` _string_      | _Required._ |
| `elementContent` _string_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_link_click_1 link_click_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'link_click'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

## Cross-domain tracking

Read more about tracking cross-domain in the [cross-navigation page](/docs/events/cross-navigation/).

---

# Media playback
> Track video and audio playback with media player events covering playback, buffering, quality changes, and ad tracking.
> Source: https://docs.snowplow.io/docs/events/ootb-data/media-events/

The Snowplow media tracking APIs enable you to track events from media playback on the web as well as in mobile apps. Track changes in the media playback e.g., play, pause, seek events, playback position with ping and percentage progress events, or ad playback events e.g., ad breaks, ad progress, or ad clicks.

While some trackers provide additional integrations with media players, the media tracking APIs are designed to be player agnostic.

Use media tracking to answer questions such as:

- Are users watching videos without sound?
- Which pieces of audio content do users play to completion?
- Where do users drop off during playback?
- How does buffering affect engagement?
- Which ads do users skip, click, or watch?

The media tracking works with a set of out-of-the-box events and entities. You can add custom entities to the events as required.

Media events fall into three categories: playback lifecycle events, player state changes, and advertising events. For most implementations, you'll need to track these events yourself.

Many media events have no properties of their own. The user behavior is captured in the event type, and the media player, session, ad, or ad break entities that are automatically attached to the event.

The event schema URIs have the format: `iglu:com.snowplowanalytics.snowplow.media/{EVENT_TYPE}/jsonschema/1-0-0`.

## Media API versions

The current set of media APIs has evolved from an earlier, more limited implementation. We refer to them as v1 and v2 of the media tracking APIs. Unless otherwise stated, all documentation refers to v2 of the media tracking APIs.

The [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt data model supports both versions since version 0.6.0. The v2 schemas are the default.

The newer v2 implementation has many more features than v1, including:

- Advertising tracking support
- Media ping events at configurable intervals
- Session entity with aggregated statistics
- Quality change tracking for bitrate, FPS, and quality level
- Buffer and seek start/end event pairs
- Filtering of repeated events (seek, volume)
- Page activity updates during playback

The v1 implementation used a single `media_player_event` schema for all events, distinguishing between actions using the `type` field. See the [Version 1 schemas](#version-1-schemas) section at the end of this page for details. The v2 implementation uses separate schemas for each event type.

## Tracker support

This table shows the support for media tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include media tracking APIs.

Auto-tracking means that the tracker can subscribe to and track events from a registered media player, without you needing to manually track each event.

| Tracker                                                                  | Plugin                                                                 | Supported | Since version                      | Auto-tracking | Notes                                                                                                                                               |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------- | --------- | ---------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Web                                                                      | [Snowplow](/docs/sources/web-trackers/tracking-events/media/snowplow/) | ✅         | 3.12.0 (Media v2)                  | ❌             |                                                                                                                                                     |
| Web                                                                      | [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/)       | ✅         | 3.2.0 (Media v1), 4.0.0 (Media v2) | ✅             |                                                                                                                                                     |
| Web                                                                      | [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/)   | ✅         | 3.2.0 (Media v1), 4.0.0 (Media v2) | ✅             |                                                                                                                                                     |
| Web                                                                      | [Vimeo](/docs/sources/web-trackers/tracking-events/media/vimeo/)       | ✅         | 3.14.0 (Media v2)                  | ✅             |                                                                                                                                                     |
| [iOS](/docs/sources/mobile-trackers/tracking-events/media-tracking/)     |                                                                        | ✅         | 5.3.0 (Media v2)                   | ✅/❌           | Auto-tracking for AVPlayer                                                                                                                          |
| [Android](/docs/sources/mobile-trackers/tracking-events/media-tracking/) |                                                                        | ✅         | 5.3.0 (Media v2)                   | ❌             |                                                                                                                                                     |
| React Native                                                             |                                                                        | ❌         |                                    |               | Use the media schemas for your own custom events, or the [media web tracking plan template](/docs/event-studio/tracking-plans/templates/#media-web) |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/media-tracking/) |                                                                        | ✅         | 0.7.0 (Media v2)                   | ❌             |                                                                                                                                                     |
| [Roku](/docs/sources/roku-tracker/media-tracking/)                       |                                                                        | ✅         | 0.1.0 (Media v1), 0.3.0 (Media v2) | ✅             |                                                                                                                                                     |
| Google Tag Manager                                                       | ❌                                                                      |           |                                    |               |                                                                                                                                                     |

The Snowplow media tracking APIs are supported by the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt data model.

We recommend using the [Media base tracking plan template](/docs/event-studio/tracking-plans/templates/#media-web) for media tracking on web.

## How to use the media APIs

Each media player instance needs to be registered with the tracker, using a unique session ID. Set the media session ID to a random string. In the browser, you could generate this with a function such as `crypto.randomUUID()`. The ID should be unique to a single session within a single player instance.

![Diagram showing media session timeline and which events to track](/assets/images/timeline_diagram-2fee2b04dd6ba2164e320661e324ae3b.svg)

> **Info:** Check out our [example React application](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media) to see tracked media events and entities live as you watch a video.
>
> The source code for the app is [available here](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

### Start a session

To start tracking user behavior involving that player, call the appropriate `startMediaTracking` function. The exact API is different for each tracker. Call `startMediaTracking` as soon as the player has loaded, not when playback begins. This ensures that session tracking begins immediately, and allows for accurate measurement of metrics such as buffering time or time to first frame.

A media session should correspond to a single media playback, including any ads that play before, during, or after the media. Metrics are collected against a media session, so it's important to set it correctly.

### Configure the player

You can provide additional player information and configuration when calling `startMediaTracking`, as well as in subsequent tracking calls.

We recommend providing a human-readable name for the media content as a `label` property. It'll be tracked as part of the media player entity. The label helps you quickly identify which content a media session relates to during analysis, and is used in combination with the media type, player type, and player ID to create the media stats table with the [Media Player dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/).

### Track playback

Track user interactions with the appropriate media tracking calls. Call `trackMediaPlay` when the main media content starts playing, even if there are pre-roll ads. The media session model accounts for both total play time and play time excluding ads.

We recommend continually updating the media player state by calling `updateMediaTracking` or equivalent `update` function **every second**. This function does not send events to the Collector, but updates internal attributes to ensure accurate metrics for the next event.

How you handle livestreams and playlists depends on whether you want to analyze them as a single session or split by content:

- **Analyze as a single session**: update the player label using `updateMediaTracking` when the content changes. This won't split the session in the data model.
- **Analyze as separate sessions per content segment**: end and restart the media session each time the content changes. Customize `media_session_id` to include segments, for example, `livestream-1`, `livestream-2` or `playlist-1`, `playlist-2`, and aggregate later using a shared prefix or a custom `livestream_id` / `playlist_id`.

### Filter events

You don't need to send the media ping events, or any other tracked media events, to the data warehouse. The media APIs include a configuration option to filter out certain events, or to keep only specified events. Filtered events aren't sent to the Collector.

To drop unwanted tracked events within the pipeline, use a [custom JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/#discard-events).

### End a session

When playback ends, call the `endMediaTracking` function to end the session, clean up event listeners, and stop any background pings. Call this when you want to reset session stats, even if the media wasn't fully watched. It does not fire any events.

On web, if the user closes the browser or tab, you don't need to call `endMediaTracking` — pings stop automatically. In single-page applications, if `endMediaTracking` is missed, you may see one or two additional ping events, up to the `maxPausedPings` limit.

There's also a `trackMediaEnd` function that you can call to track when the media reaches 100% progress (fully watched). This is tracked as a media event. It's common to see sessions without a `trackMediaEnd` event if the user didn't finish the content.

If the content finishes and the user rewinds, continue the same session. Use the `timePlayed` and `contentWatched` metrics within the media session entity to understand rewatches compared to how much of the media content they watched.

## Playback lifecycle events

These events track the core playback flow from start to finish.

### Media pings

Media ping events are events sent in a regular interval while media tracking is active. They inform about the current state of the media playback, and keep the media session alive.

By default, ping events are sent every 30 seconds. They're sent in an interval that is unrelated to the media playback. However, to prevent sending too many events, there is a limit to how many ping events can be sent while the media is paused. By default, this is set to 1. You can change this by configuring `maxPausedPings` (not available on Roku).

### `ping_event`

**Type:** Event

Media player event fired periodically during main content playback, regardless of other API events that have been sent.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ping_event_1 ping_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ping_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Buffer end

Track a buffering end event when the the player finishes buffering content and resumes playback.

### `buffer_end_event`

**Type:** Event

Media player event fired when the the player finishes buffering content and resumes playback.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_buffer_end_event_1 buffer_end_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'buffer_end_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Buffer start

Track a buffering start event when the player goes into the buffering state and begins to buffer content.

The tracker will calculate the time since this event until a buffer end event, play event, or a change in playback position. It will add the duration to the `timeBuffering` property in the media session entity.

### `buffer_start_event`

**Type:** Event

Media player event fired when the player goes into the buffering state and begins to buffer content.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_buffer_start_event_1 buffer_start_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'buffer_start_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Pause event

Track a pause event when the user pauses the playback.

Tracking this event will automatically set the `paused` property in the media player entity to `true`.

### `pause_event`

**Type:** Event

Media player event sent when the user pauses the playback.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_pause_event_1 pause_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'pause_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Playback end

Track a playback end event when playback stops, either when the end of the media is reached, or because no further data is available.

Tracking this event will automatically set the `ended` and `paused` properties in the media player entity to `true`.

### `end_event`

**Type:** Event

Media player event sent when playback stops when end of the media is reached or because no further data is available.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_end_event_1 end_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'end_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Percent progress

The tracker will track percentage progress events automatically, when the playback reaches configured percentage boundaries.

### `percent_progress_event`

**Type:** Event

Media player event fired when a percentage boundary set in tracking options is reached

**Properties:**

| Property                    | Description                                          |
| --------------------------- | ---------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the media |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_percent_progress_event_1 percent_progress_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'percent_progress_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Play

Track a play event when the player changes state to playing from previously being paused.

Tracking this event will automatically set the `paused` property in the media player entity to `false`.

### `play_event`

**Type:** Event

Media player event sent when the player changes state to playing from previously being paused.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_play_event_1 play_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'play_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Seek end

Track a seek end event when a seek operation completes.

### `seek_end_event`

**Type:** Event

Media player event sent when a seek operation completes.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_seek_end_event_1 seek_end_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'seek_end_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Seek start

Track a seek start event when a seek operation begins.

### `seek_start_event`

**Type:** Event

Media player event sent when a seek operation begins.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_seek_start_event_1 seek_start_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'seek_start_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

## Player state change events

These events track changes to the player's display mode, quality, and settings.

### Error

Track an error event when the resource couldn't be loaded due to an error.

### `error_event`

**Type:** Event

Media player event tracked when the resource could not be loaded due to an error.

**Properties:**

| Property                    | Description                                                                          |
| --------------------------- | ------------------------------------------------------------------------------------ |
| `errorCode` _string_        | _Optional._ Error-identifying code for the playback issue. E.g. E522                 |
| `errorName` _string_        | _Optional._ Name for the type of error that occurred in the playback. E.g. forbidden |
| `errorDescription` _string_ | _Optional._ Longer description for the error that occurred in the playback.          |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_error_event_1 error_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'error_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Fullscreen change

Track a fullscreen change event when the media player fullscreen changes, fired immediately after the browser switches into or out of full-screen mode.

The `fullscreen` value is passed when tracking the event, and is automatically updated in the `player` entity.

### `fullscreen_change_event`

**Type:** Event

Media player event fired immediately after the browser switches into or out of full-screen mode.

**Properties:**

| Property               | Description                                                          |
| ---------------------- | -------------------------------------------------------------------- |
| `fullscreen` _boolean_ | _Required._ Whether the video element is fullscreen after the change |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_fullscreen_change_event_1 fullscreen_change_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'fullscreen_change_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Picture-in-picture change

Track a picture-in-picture change event immediately after the platform switches into or out of picture-in-picture mode.

The `pictureInPicture` value is passed when tracking the event, and is automatically updated in the `player` entity.

### `picture_in_picture_change_event`

**Type:** Event

Media player event fired immediately after the browser switches into or out of picture-in-picture mode.

**Properties:**

| Property                     | Description                                                                           |
| ---------------------------- | ------------------------------------------------------------------------------------- |
| `pictureInPicture` _boolean_ | _Required._ Whether the video element is showing picture-in-picture after the change. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_picture_in_picture_change_event_1 picture_in_picture_change_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'picture_in_picture_change_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Playback rate change

Track a playback rate change event when the playback rate has changed.

The `previousRate` is set automatically based on the last `playbackRate` value in the `player` entity. The `newRate` is passed when tracking the event, and is automatically updated in the `player` entity.

### `playback_rate_change_event`

**Type:** Event

Media player event sent when the playback rate has changed.

**Properties:**

| Property                | Description                                               |
| ----------------------- | --------------------------------------------------------- |
| `previousRate` _number_ | _Optional._ Playback rate before the change (1 is normal) |
| `newRate` _number_      | _Required._ Playback rate after the change (1 is normal)  |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_playback_rate_change_event_1 playback_rate_change_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'playback_rate_change_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Quality change

Track a quality change event when the video playback quality changes.

The `previousQuality` is set automatically based on the last `quality` value in the `player` entity. The `newQuality` is passed when tracking the event, and is automatically updated in the `player` entity.

### `quality_change_event`

**Type:** Event

Media player event tracked when the video playback quality changes automatically.

**Properties:**

| Property                    | Description                                                            |
| --------------------------- | ---------------------------------------------------------------------- |
| `previousQuality` _string_  | _Optional._ Quality level before the change (e.g., 1080p).             |
| `newQuality` _string_       | _Optional._ Quality level after the change (e.g., 1080p).              |
| `bitrate` _integer_         | _Optional._ The current bitrate in bits per second.                    |
| `framesPerSecond` _integer_ | _Optional._ The current number of frames per second.                   |
| `automatic` _boolean_       | _Optional._ Whether the change was automatic or triggered by the user. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_quality_change_event_1 quality_change_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'quality_change_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ready

Track a ready event when the media tracking is successfully attached to the player and can track events.

### `ready_event`

**Type:** Event

Media player event fired when the media tracking is successfully attached to the player and can track events.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ready_event_1 ready_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ready_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Volume change

Track a volume change event when the user changes the volume.

The `previousVolume` is set automatically based on the last `volume` value in the `player` entity. The `newVolume` is passed when tracking the event, and is automatically updated in the `player` entity.

### `volume_change_event`

**Type:** Event

Media player event sent when the volume has changed.

**Properties:**

| Property                   | Description                                      |
| -------------------------- | ------------------------------------------------ |
| `previousVolume` _integer_ | _Optional._ Volume percentage before the change. |
| `newVolume` _integer_      | _Required._ Volume percentage after the change.  |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_volume_change_event_1 volume_change_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'volume_change_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

## Advertising events

These events track ad playback for monetized content. Ad events include the media ad and media ad break entities.

Tracking only `trackMediaAdStart` and `trackMediaAdComplete`, or equivalent API, is sufficient if you don't need to attribute ads to specific ad breaks.

If you skip `trackMediaAdBreakStart` and `trackMediaAdBreakEnd`, you won't have details like pod size and break ID to analyze all the ads within a break.

### Ad click

Track an ad click event when a user clicks on an ad.

Tracking this event will increase the counter of `adsClicked` in the session entity.

### `ad_click_event`

**Type:** Event

Media player event fired when the user clicked on the ad.

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_click_event_1 ad_click_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_click_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad break start

Track an ad break start event at the start of an ad break.

Tracking this event will increase the counter of `adBreaks` in the media session entity.

### `ad_break_start_event`

**Type:** Event

Media player event that signals the start of an ad break.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_break_start_event_1 ad_break_start_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_break_start_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad break end

Track an ad break end event to signal the end of an ad break.

### `ad_break_end_event`

**Type:** Event

Media player event that signals the end of an ad break.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_break_end_event_1 ad_break_end_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_break_end_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad complete

Track an ad complete event to signal that the ad creative was played to the end, at normal speed.

### `ad_complete_event`

**Type:** Event

Media player event that signals the ad creative was played to the end at normal speed.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_complete_event_1 ad_complete_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_complete_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad pause

Track an ad pause event when a user clicks the pause control to stop the ad creative.

### `ad_pause_event`

**Type:** Event

Media player event fired when the user clicked the pause control and stopped the ad creative.

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_pause_event_1 ad_pause_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_pause_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad quartile

The trackers have dedicated functions to track each ad quartile event: `trackMediaAdFirstQuartile`, `trackMediaAdMidpoint`, `trackMediaAdThirdQuartile`, or equivalent API. These events set the `percentProgress` property automatically to 25%, 50%, and 75% respectively.

### `ad_quartile_event`

**Type:** Event

Media player event fired when a quartile of ad is reached after continuous ad playback at normal speed.

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Required._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_quartile_event_1 ad_quartile_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_quartile_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad resume

Track an ad resume event when a user resumes playing the ad creative after it had been stopped or paused.

### `ad_resume_event`

**Type:** Event

Media player event fired when the user resumed playing the ad creative after it had been stopped or paused.

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_resume_event_1 ad_resume_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_resume_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad skip

Track an ad skip event when the user activates a skip control to skip the ad creative.

Tracking this event will increase the counter of `adsSkipped` in the media session entity.

### `ad_skip_event`

**Type:** Event

Media player event fired when the user activated a skip control to skip the ad creative.

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_skip_event_1 ad_skip_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_skip_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad start

Track an ad start event at the start of an ad.

Tracking this event will increase the counter of `ads` in the media session entity.

### `ad_start_event`

**Type:** Event

Media player event that signals the start of an ad.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_ad_start_event_1 ad_start_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'ad_start_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

## Custom events

Track your own custom events within the media session, using `trackMediaSelfDescribingEvent` or equivalent API. The tracker will automatically attach the media entities.

## Automatically included entities

Every media event includes entities that describe the current state of the player and session. These entities are attached automatically by the tracker. You don't need to track them yourself.

| Entity         | Attached to      | Purpose                                                      |
| -------------- | ---------------- | ------------------------------------------------------------ |
| Media player   | All media events | Current playback state (position, duration, volume, quality) |
| Media session  | All media events | Aggregated session metrics (time played, content watched)    |
| Media ad       | Ad events only   | Information about the current ad                             |
| Media ad break | Ad events only   | Information about the ad break                               |

You can also add custom entities to all media events, by providing them in the configuration. These entities will be attached to all subsequent media events for that player instance.

As with all Snowplow events, you can also attach custom entities to individual media events when tracking them.

### Media player

The media player entity captures the current state of playback at the moment each event fires. It's required for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model.

### `media_player`

**Type:** Schema

Schema for a context entity for media events that describes the media player and playback state

**Properties:**

| Property                     | Description                                                                                                   |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `currentTime` _number_       | _Required._ The current playback time position within the media in seconds.                                   |
| `duration` _number_          | _Optional._ A floating-point value indicating the duration of the media in seconds.                           |
| `ended` _boolean_            | _Required._ Whether playback of the media has ended.                                                          |
| `fullscreen` _boolean_       | _Optional._ Whether the video element is fullscreen.                                                          |
| `livestream` _boolean_       | _Optional._ Whether the media is a live stream.                                                               |
| `label` _string_             | _Optional._ Human readable name given to tracked media content.                                               |
| `loop` _boolean_             | _Optional._ Whether the video should restart after ending.                                                    |
| `mediaType` _string_         | _Optional._ Type of media content. Must be one of: `video`, `audio`,``                                        |
| `muted` _boolean_            | _Optional._ Whether the media element is muted.                                                               |
| `paused` _boolean_           | _Required._ Whether the media element is paused                                                               |
| `pictureInPicture` _boolean_ | _Optional._ Whether the video element is showing picture-in-picture.                                          |
| `playbackRate` _number_      | _Optional._ Playback rate (1 is normal).                                                                      |
| `playerType` _string_        | _Optional._ Type of the media player (e.g., com.youtube-youtube, com.vimeo-vimeo, org.whatwg-media\_element). |
| `quality` _string_           | _Optional._ Quality level of the playback (e.g., 1080p).                                                      |
| `volume` _integer_           | _Optional._ Volume percentage at which the media will be played.                                              |

### Media session

The media session entity contains metrics calculated based on the tracked media events and the media update calls. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model.

The table below shows which media player properties and media events are used to calculate the metrics within the media session entity.

| Media player entity property | Media events                                               | Affected calculation of metric                                  |
| ---------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------- |
| `paused`                     | `play_event`, `pause_event`                                | `timePlayed`, `timePaused`, `timePlayedMuted`, `contentWatched` |
| `currentTime`                |                                                            | `timePlayed`, `timePaused`, `timePlayedMuted`, `contentWatched` |
| `muted`                      |                                                            | `timePlayedMuted`                                               |
| `playbackRate`               | `playback_rate_change_event`                               | `avgPlaybackRate`                                               |
|                              | `buffer_start_event`, `buffer_end_event`, `play_event`     | `timeBuffering`                                                 |
|                              | `ad_start_event`                                           | `ads`                                                           |
|                              | `ad_skip_event`                                            | `adsSkipped`                                                    |
|                              | `ad_click_event`                                           | `adsClicked`                                                    |
|                              | `ad_break_start_event`                                     | `adBreaks`                                                      |
|                              | `ad_start_event`, `ad_quartile_event`, `ad_complete_event` | `timeSpentAds`                                                  |
|                              | `ad_start_event`, `ad_complete_event`, `ad_skip_event`     | `timePlayed`, `timePlayedMuted`\*                               |

\* Play time stats aren't incremented while ads with type linear (default) are being played. Linear ads take over the video playback. For non-linear and companion ads, play time stats are still incremented while the ad is playing.

### `session`

**Type:** Schema

Schema for a context entity for media player events that tracks a session of a single media player usage

**Properties:**

| Property                   | Description                                                                                                                                                                                                   |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mediaSessionId` _string_  | _Required._ An identifier for the media session that is kept while the media content is played in the media player.                                                                                           |
| `startedAt` _null_         | _Optional._ Local date-time timestamp of when the session started.                                                                                                                                            |
| `pingInterval` _integer_   | _Optional._ Interval (seconds) in which the ping events will be sent. Default (specified in the tracker media docs) is assumed if not specified.                                                              |
| `timePlayed` _number_      | _Optional._ Total seconds user spent playing content (excluding linear ads).                                                                                                                                  |
| `timePlayedMuted` _number_ | _Optional._ Total seconds user spent playing content on mute (excluding linear ads).                                                                                                                          |
| `timePaused` _number_      | _Optional._ Total seconds user spent with paused content (excluding linear ads).                                                                                                                              |
| `contentWatched` _number_  | _Optional._ Total seconds of the content played. Each part of the content played is counted once (i.e., counts rewinding or rewatching the same content only once). Playback rate does not affect this value. |
| `timeBuffering` _number_   | _Optional._ Total seconds that playback was buffering during the session.                                                                                                                                     |
| `timeSpentAds` _number_    | _Optional._ Total seconds that ads played during the session.                                                                                                                                                 |
| `ads` _integer_            | _Optional._ Number of ads played.                                                                                                                                                                             |
| `adsClicked` _integer_     | _Optional._ Number of ads that the user clicked on.                                                                                                                                                           |
| `adsSkipped` _integer_     | _Optional._ Number of ads that the user skipped.                                                                                                                                                              |
| `adBreaks` _integer_       | _Optional._ Number of ad breaks played.                                                                                                                                                                       |
| `avgPlaybackRate` _number_ | _Optional._ Average playback rate (1 is normal speed).                                                                                                                                                        |

### Media ad

The media ad entity describes the currently playing ad. It's attached to ad events.

### `ad`

**Type:** Schema

Schema for a context entity with information about the currently played ad

**Properties:**

| Property                | Description                                                                      |
| ----------------------- | -------------------------------------------------------------------------------- |
| `name` _null_           | _Optional._ Friendly name of the ad.                                             |
| `adId` _string_         | _Required._ Unique identifier for the ad.                                        |
| `creativeId` _string_   | _Optional._ The ID of the ad creative.                                           |
| `podPosition` _integer_ | _Optional._ The position of the ad within the ad break, starting with 1.         |
| `duration` _number_     | _Optional._ Length of the video ad in seconds.                                   |
| `skippable` _boolean_   | _Optional._ Indicating whether skip controls are made available to the end user. |

### Media ad break

The media ad break entity describes a group of ads played together, whether pre-roll, mid-roll, or post-roll. It is attached to ad break events and all ad events within the break.

### `ad_break`

**Type:** Schema

Schema for a context entity, shared with all ad events belonging to the ad break.

**Properties:**

| Property             | Description                                                                                                                                                                                                                                                             |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` _null_        | _Optional._ Ad break name (e.g., pre-roll, mid-roll, and post-roll).                                                                                                                                                                                                    |
| `breakId` _string_   | _Required._ An identifier for the ad break.                                                                                                                                                                                                                             |
| `startTime` _number_ | _Required._ Playback time in seconds at the start of the ad break.                                                                                                                                                                                                      |
| `breakType` _string_ | _Optional._ Type of ads within the break: linear (take full control of the video for a period of time), nonlinear (run concurrently to the video), companion (accompany the video but placed outside the player). Must be one of: `linear`, `nonlinear`, `companion`,`` |
| `podSize` _integer_  | _Optional._ The number of ads to be played within the ad break.                                                                                                                                                                                                         |

## Player-specific data

The [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/), [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/), and [Vimeo](/docs/sources/web-trackers/tracking-events/media/vimeo/) media plugins automatically attach additional entities specific to those players. The Vimeo plugin also tracks additional events. See the documentation for each plugin for details.

The HTML5 and YouTube entities are used as part of the v1 media schemas for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt package. If you're using the v2 schemas (default from Media Player 0.6+), the Media Player model doesn't use any fields from these player-specific entities. You can still use them for your own analysis.

The Vimeo events and entities aren't used by any version of the Media Player model.

## Version 1 schemas

The v1 media events use the `media_player_event` schema.

### `media_player_event`

**Type:** Event

Schema for a media event

**Properties:**

| Property         | Description                                     |
| ---------------- | ----------------------------------------------- |
| `type` _string_  | _Required._ The event fired by the media player |
| `label` _string_ | _Optional._ A custom identifier                 |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_media_player_event_1 media_player_event_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'media_player_event'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

The v1 media player entity uses the `1-0-0` version of the `media_player` schema, while v2 uses `2-0-0`.

### `media_player`

**Type:** Schema

Common Context Schema for a media player event

**Properties:**

| Property                    | Description                                                                                         |
| --------------------------- | --------------------------------------------------------------------------------------------------- |
| `currentTime` _number_      | _Required._ The current playback time                                                               |
| `duration` _number_         | _Required._ A double-precision floating-point value indicating the duration of the media in seconds |
| `ended` _boolean_           | _Required._ If playback of the media has ended                                                      |
| `isLive` _boolean_          | _Optional._ If the media is live                                                                    |
| `loop` _boolean_            | _Required._ If the video should restart after ending                                                |
| `muted` _boolean_           | _Required._ If the media element is muted                                                           |
| `paused` _boolean_          | _Required._ If the media element is paused                                                          |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the media                                                |
| `playbackRate` _number_     | _Required._ Playback rate (1 is normal)                                                             |
| `volume` _integer_          | _Required._ Volume percent                                                                          |

---

# Mobile app install and lifecycle events
> Automatically track mobile app install, foreground, and background events with lifecycle entities for app usage analysis.
> Source: https://docs.snowplow.io/docs/events/ootb-data/mobile-lifecycle-events/

Mobile lifecycle events track key moments in your app's usage: when a user installs the app, opens it, or switches away. These events help you understand user engagement patterns, measure retention, and analyze session behavior.

We recommend using the [Base mobile tracking plan template](/docs/event-studio/tracking-plans/templates/#base-mobile) for mobile tracking. It includes install and lifecycle events.

## Install events

The mobile trackers can automatically track an install event when the app is first opened after installation.

This table shows the support for mobile application install tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). Depending on your tracker and version, install autotracking may be enabled by default.

| Tracker                                                                                   | Supported | Since version | Auto-tracking | Notes                           |
| ----------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------- |
| Web                                                                                       | ❌         |               |               |                                 |
| [iOS](/docs/sources/mobile-trackers/tracking-events/installation-tracking/)               | ✅         | 1.1.0         | ✅             |                                 |
| [Android](/docs/sources/mobile-trackers/tracking-events/installation-tracking/)           | ✅         | 1.1.0         | ✅             |                                 |
| [React Native](/docs/sources/react-native-tracker/tracking-events/installation-tracking/) | ✅         | 0.1.0         | ✅             | Only relevant for mobile events |
| Flutter                                                                                   | ❌         |               |               |                                 |
| Roku                                                                                      | ❌         |               |               |                                 |
| Google Tag Manager                                                                        | ❌         |               |               |                                 |

> **Note:** It's not possible to track when an app is uninstalled, since the mobile platforms don't provide a callback where such an event could be tracked.

This event has no properties since the relevant data is the timestamp, which is captured in the standard event fields.

### `application_install`

**Type:** Event

Schema for an event where a mobile application is installed.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_mobile_application_install_1 application_install_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'application_install'

and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Android referrer details entity

On Android only, the Android tracker can attach an extra [entity with install referrer information](/docs/sources/mobile-trackers/tracking-events/installation-tracking/#android-only-tracking-referrer-information-from-the-google-play-referrer-library) to the install event. It makes use of the [Google Play Install Referrer library](https://developer.android.com/google/play/installreferrer) to retrieve the referrer information.

This entity is available from Android tracker version 5.2.0 onwards.

### `referrer_details`

**Type:** Event

Represents an install referrer details for Android apps installed from the Play Store (see https\://developer.android.com/reference/com/android/installreferrer/api/ReferrerDetails)

**Properties:**

| Property                           | Description                                                                                                    |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `installReferrer` _string_         | _Optional._ The referrer URL of the installed package                                                          |
| `referrerClickTimestamp` _string_  | _Optional._ The timestamp when referrer click happens                                                          |
| `installBeginTimestamp` _string_   | _Optional._ The timestamp when installation begins                                                             |
| `googlePlayInstantParam` _boolean_ | _Required._ Boolean indicating if the user has interacted with the app's instant experience in the past 7 days |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_android_installreferrer_api_referrer_details_1 referrer_details_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'referrer_details'

and events.event_vendor = 'com.android.installreferrer.api'
```

## Foreground and background

Foreground and background events capture when the user switches to or away from your app. A foreground event fires when the app becomes visible on screen, while a background event fires when the user switches to another app or returns to the home screen.

The mobile trackers can track these events automatically. They also attach a lifecycle entity to all events, indicating whether the app was visible when each event occurred.

This table shows the support for mobile lifecycle event tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include lifecycle event tracking. Depending on your tracker and version, lifecycle autotracking may be enabled by default.

| Tracker                                                                                                | Supported | Since version | Auto-tracking | Notes                           |
| ------------------------------------------------------------------------------------------------------ | --------- | ------------- | ------------- | ------------------------------- |
| Web                                                                                                    | ❌         |               |               |                                 |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                              | ✅         | 3.0.0         | ✅             |                                 |
| [Android](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                          | ✅         | 3.0.0         | ✅             |                                 |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-deep-link-received-events) | ✅         | 0.1.0         | ✅             | Only relevant for mobile events |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                             | ✅         | 0.5.0         | ✅             | Only relevant for mobile events |
| Roku                                                                                                   | ❌         |               |               |                                 |
| Google Tag Manager                                                                                     | ❌         |               |               |                                 |

### Foreground event

The foreground event is tracked each time the app becomes visible on the screen. The `foregroundIndex` property counts how many times the app has been foregrounded since installation.

### `application_foreground`

**Type:** Event

Schema for an application foreground event

**Properties:**

| Property                    | Description |
| --------------------------- | ----------- |
| `foregroundIndex` _integer_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_application_foreground_1 application_foreground_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'application_foreground'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Background event

The background event is tracked when the app is no longer visible, such as when the user switches to another app or returns to the home screen. The `backgroundIndex` property counts how many times the app has been backgrounded since installation.

### `application_background`

**Type:** Event

Schema for an application background event

**Properties:**

| Property                    | Description |
| --------------------------- | ----------- |
| `backgroundIndex` _integer_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_application_background_1 application_background_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'application_background'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Lifecycle entity

When lifecycle autotracking is enabled, this entity is automatically attached to all events. It indicates whether the app was in the foreground or background when the event occurred.

### `application_lifecycle`

**Type:** Schema

Entity that indicates the visibility state of the app (foreground, background)

**Properties:**

| Property              | Description                                                                                                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `isVisible` _boolean_ | _Required._ Indicates if the app is in foreground state (true) or background state (false)                                                                               |
| `index` _integer_     | _Optional._ Represents the foreground index or background index (tracked with com.snowplowanalytics.snowplow application\_foreground and application\_background events. |

---

# Page and screen engagement
> Measure user engagement time and scrolling depth using page ping events on web and screen summary entities on mobile.
> Source: https://docs.snowplow.io/docs/events/ootb-data/page-activity-tracking/

Activity or engagement tracking enables you to measure the time a user spent engaged on a page or screen, and the extent of the page or screen they viewed.

Snowplow provides implementations for web and mobile platforms:

- Page ping events on web
- Screen summary entity on mobile

Both page and screen activity tracking are included in the [Snowplow Unified Digital package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) to calculate engagement metrics. The legacy [Web package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) handles page pings as well.

## Page engagement

Page ping events record users engaging with content on a web page after it has initially loaded. This includes scroll position.

Where activity autotracking is available, you can configure your tracker to send page ping events at regular intervals. They're considered a type of "heartbeat" event that indicates the user is still present and engaging with the page.

Page ping events are [baked-in events](/docs/fundamentals/events/) that have no schema. They populate the [page offset atomic event parameters](/docs/fundamentals/canonical-event/#baked-in-event-fields).

This table shows the support for page ping tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). Python is the only server-side tracker that provides page ping events.

| Tracker                                                                    | Supported | Since version | Auto-tracking | Notes                              |
| -------------------------------------------------------------------------- | --------- | ------------- | ------------- | ---------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/activity-page-pings/)     | ✅         | 0.10.0        | ✅             |                                    |
| iOS                                                                        | ❌         |               |               |                                    |
| Android                                                                    | ❌         |               |               |                                    |
| React Native                                                               | ❌         |               |               |                                    |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/) | ✅         | 0.1.0         | ✅             | Only on web                        |
| Roku                                                                       | ❌         |               |               |                                    |
| [Python](/docs/sources/python-tracker/tracking-specific-events/)           | ✅         | 0.8.0         | ❌             | Detect activity and track manually |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)  | ✅         | v3            | ✅             |                                    |

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes page pings.

### Activity metrics entity

Since JavaScript tracker version 4.8.0, you can also attach [detailed activity metrics](/docs/sources/web-trackers/tracking-events/activity-page-pings/#activity-metrics) to each page ping. The detailed metrics show how much interaction happened on the page, including the number of clicks, touches, key presses, absolute scroll distance (in either direction), and distance traveled by the mouse.

> **Tip:** This can be helpful, for example, for distinguishing human and agent actions in an agentic browser session. Agents typically produce very little activity, navigating to pages directly and not clicking on links.

### `activity_metrics`

**Type:** Schema

Quantitative user interaction metrics accumulated between page pings

**Properties:**

| Property                   | Description                                                                 |
| -------------------------- | --------------------------------------------------------------------------- |
| `mouseDistance` _integer_  | _Required._ Cumulative Euclidean mouse distance in pixels                   |
| `scrollDistance` _integer_ | _Required._ Cumulative absolute scroll distance in pixels (both directions) |
| `keyPresses` _integer_     | _Required._ Number of key presses                                           |
| `clicks` _integer_         | _Required._ Number of clicks                                                |
| `touches` _integer_        | _Required._ Number of touch events                                          |

## Screen engagement

Use screen engagement tracking to track a `screen_end` event when a user navigates away from a screen.

The trackers can also track a `screen_summary` entity that contains screen engagement data. This entity is sent along with [lifecycle](/docs/events/ootb-data/mobile-lifecycle-events/) and `screen_end` events.

This table shows the support for screen engagement tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include screen engagement tracking.

| Tracker                                                                                                        | Supported | Since version | Auto-tracking | Notes                           |
| -------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/screen-views/#screen-engagement-tracking)                     | ✅         | 4.2.0         | ✅/❌           | Requires screen tracking plugin |
| [iOS](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking)               | ✅         | 6.0.0         | ✅             |                                 |
| [Android](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking)           | ✅         | 6.0.0         | ✅             |                                 |
| [React Native](/docs/sources/react-native-tracker/tracking-events/screen-tracking/#screen-engagement-tracking) | ✅         | 2.1.0         | ✅             | Only on mobile                  |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/#screen-engagement-tracking)                           | ✅         | 0.1.0         | ✅             | Only on mobile                  |
| Roku                                                                                                           | ❌         |               |               |                                 |
| Google Tag Manager                                                                                             | ❌         |               |               |                                 |

We recommend using the [Base mobile tracking plan template](/docs/event-studio/tracking-plans/templates/#base-mobile) for mobile tracking. It includes screen end events.

### Screen end event

This event can be tracked automatically just before the transition to the next screen, based on the firing of the next screen view event. It has no properties.

### `screen_end`

**Type:** Event

Schema for an event tracked before transitioning to a new screen

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_mobile_screen_end_1 screen_end_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'screen_end'

and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### List item view event

Track this event to mark which items are visible in a list. When screen engagement tracking is enabled, this event won't be sent to the Collector but used to update the tracker state for the screen summary entity.

If you're using the iOS tracker with SwiftUI, you can configure it to [automatically track this event](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#list-item-view-tracking).

### `list_item_view`

**Type:** Event

Schema for an event tracked when an item is displayed in a list

**Properties:**

| Property                | Description                                               |
| ----------------------- | --------------------------------------------------------- |
| `index` _integer_       | _Required._ Index of the item in a list on the screen     |
| `items_count` _integer_ | _Optional._ Total number of items in a list on the screen |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_mobile_list_item_view_1 list_item_view_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'list_item_view'

and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Scroll changed event

Track this event to mark changes in scroll position on a screen. When screen engagement tracking is enabled, this event won't be sent to the Collector but used to update the tracker state for the screen summary entity.

### `scroll_changed`

**Type:** Event

Schema for an event tracked when a scroll view's scroll position changes

**Properties:**

| Property                   | Description                                                        |
| -------------------------- | ------------------------------------------------------------------ |
| `x_offset` _integer_       | _Optional._ Horizontal scroll offset in pixels                     |
| `y_offset` _integer_       | _Optional._ Vertical scroll offset in pixels                       |
| `view_width` _integer_     | _Optional._ The width of the scroll view in pixels                 |
| `view_height` _integer_    | _Optional._ The height of the scroll view in pixels                |
| `content_width` _integer_  | _Optional._ The width of the content in the scroll view in pixels  |
| `content_height` _integer_ | _Optional._ The height of the content in the scroll view in pixels |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_mobile_scroll_changed_1 scroll_changed_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'scroll_changed'

and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Screen summary entity

Entity that contains screen engagement information, including how long a user spends on a screen in the foreground and background, as well as information about scroll depth and list items.

The entity data is based on tracker state captured from lifecycle, list item view, and scroll changed events.

### `screen_summary`

**Type:** Schema

Schema for an entity tracked with foreground/background/screen\_end events with summary statistics about the screen view

**Properties:**

| Property                    | Description                                                                             |
| --------------------------- | --------------------------------------------------------------------------------------- |
| `foreground_sec` _number_   | _Required._ Time in seconds spent on the current screen while the app was in foreground |
| `background_sec` _number_   | _Optional._ Time in seconds spent on the current screen while the app was in background |
| `last_item_index` _integer_ | _Optional._ Index of the last viewed item in the list on the screen                     |
| `items_count` _integer_     | _Optional._ Total number of items in the list on the screen                             |
| `min_x_offset` _integer_    | _Optional._ Minimum horizontal scroll offset on the scroll view in pixels               |
| `max_x_offset` _integer_    | _Optional._ Maximum horizontal scroll offset on the scroll view in pixels               |
| `min_y_offset` _integer_    | _Optional._ Minimum vertical scroll offset on the scroll view in pixels                 |
| `max_y_offset` _integer_    | _Optional._ Maximum vertical scroll offset on the scroll view in pixels                 |
| `content_width` _integer_   | _Optional._ Width of the scroll view in pixels                                          |
| `content_height` _integer_  | _Optional._ Height of the scroll view in pixels                                         |

---

# Page and screen views
> Track page views on web and screen views on mobile with page view IDs and screen context entities.
> Source: https://docs.snowplow.io/docs/events/ootb-data/page-and-screen-view-events/

Page and screen view tracking is key for understanding your users' navigation within your app.

Snowplow provides implementations for web and mobile platforms:

- Page view events on web
- Screen view events on mobile

Both page and screen views are included in the [Snowplow Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/). The legacy [Web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) and [Mobile](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/) packages handle page and screen views respectively.

## Page views

Track page views to record when a user views a web page.

Page view events are [baked-in events](/docs/fundamentals/events/) that have no schema. They populate the `page` and `url` [atomic event parameters](/docs/fundamentals/canonical-event/#page-fields).

Check out the [Links and referrers](/docs/events/ootb-data/links-and-referrers/) overview for information about the `refr` or `page_referrer` atomic event property.

This table shows the support for page view tracking across the main [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                                                       | Supported | Since version | Auto-tracking | Notes                                                   |
| --------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/page-views/)                                 | ✅         | 0.1.0         | ✅/❌           | Web page entity can be automatically added              |
| iOS                                                                                           | ❌         |               |               |                                                         |
| Android                                                                                       | ❌         |               |               |                                                         |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-page-view-events) | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                    | ✅         | 0.1.0         | ✅/❌           | Only on web; web page entity can be automatically added |
| Roku                                                                                          | ❌         |               |               |                                                         |
| Node.js                                                                                       | ❌         |               |               |                                                         |
| Golang                                                                                        | ❌         |               |               |                                                         |
| .NET                                                                                          | ❌         |               |               |                                                         |
| [Java](/docs/sources/java-tracker/tracking-events/)                                           | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| [Python](/docs/sources/python-tracker/tracking-specific-events/)                              | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| Scala                                                                                         | ❌         |               |               |                                                         |
| [Ruby](/docs/sources/ruby-tracker/tracking-events/)                                           | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| Rust                                                                                          | ❌         |               |               |                                                         |
| PHP                                                                                           | ❌         |               |               |                                                         |
| C++                                                                                           | ❌         |               |               |                                                         |
| Unity                                                                                         | ❌         |               |               |                                                         |
| Lua                                                                                           | ❌         |               |               |                                                         |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)                     | ✅         | v3            | ❌             |                                                         |

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes page views.

### Web page entity

Some trackers assign a unique identifier to every page load: the page view ID. Where available, you can configure this UUID to be added to every event tracked on that page, by tracking the `web_page` entity. This identifier can be very useful for sessionization and user journey analysis.

### `web_page`

**Type:** Schema

Schema for a web page context

**Properties:**

| Property      | Description |
| ------------- | ----------- |
| `id` _string_ | _Required._ |

## Screen views

Screen view events are the mobile equivalent of page view events.

This table shows the support for screen view tracking across the main [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                                                                                         |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/screen-views/)           | ✅         | 4.2.0         | ✅/❌           | Requires screen tracking plugin; we recommend using page views instead                        |
| [iOS](/docs/sources/mobile-trackers/tracking-events/screen-tracking/)     | ✅         | 0.1.0         | ✅/❌           | Screen entity can be automatically added; screen view autotracking depends on platform        |
| [Android](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) | ✅         | 0.1.0         | ✅/❌           | Screen entity can be automatically added; screen view autotracking depends on platform        |
| [React Native](/docs/sources/react-native-tracker/tracking-events/)       | ✅         | 0.1.0         | ✅/❌           | Screen entity can be automatically added; add navigation handler for screen view autotracking |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/)                 | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Roku](/docs/sources/roku-tracker/tracking-events/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Node.js](/docs/sources/node-js-tracker/tracking-events/)                 | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Golang](/docs/sources/golang-tracker/)                                   | ✅         | 1.0.0         | ❌             | No screen entity                                                                              |
| [.NET](/docs/sources/net-tracker/)                                        | ✅         | 1.3.0         |               | No screen entity                                                                              |
| [Java](/docs/sources/java-tracker/tracking-events/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Python](/docs/sources/python-tracker/tracking-specific-events/)          | ✅         | 0.2.0         | ❌             | No screen entity                                                                              |
| Scala                                                                     | ❌         |               |               |                                                                                               |
| [Ruby](/docs/sources/ruby-tracker/tracking-events/)                       | ✅         | 0.2.0         | ❌             | No screen entity                                                                              |
| [Rust](/docs/sources/rust-tracker/tracking-events/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [PHP](/docs/sources/php-tracker/tracking-an-event/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [C++](/docs/sources/c-tracker/tracking-specific-events/)                  | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Unity](/docs/sources/unity-tracker/)                                     | ✅         | 0.8.0         | ❌             | No screen entity                                                                              |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| Google Tag Manager                                                        | ❌         |               |               |                                                                                               |

We recommend using the [Base mobile tracking plan template](/docs/event-studio/tracking-plans/templates/#base-mobile) for mobile tracking. It includes screen views.

Screen view events record the name and identifier of the current screen, as well as some optional properties such as the ID of the previous screen. Information about the previous screen can be useful for understanding user navigation within your application.

### `screen_view`

**Type:** Event

Schema for a screen view event

**Properties:**

| Property                  | Description                                                             |
| ------------------------- | ----------------------------------------------------------------------- |
| `name` _string_           | _Required._ The name of the screen viewed.                              |
| `type` _string_           | _Optional._ The type of screen that was viewed e.g feed / carousel.     |
| `id` _string_             | _Required._ An ID from the associated screenview event.                 |
| `previousName` _string_   | _Optional._ The name of the previous screen.                            |
| `previousId` _string_     | _Optional._ A screenview ID of the previous screenview.                 |
| `previousType` _string_   | _Optional._ The screen type of the previous screenview.                 |
| `transitionType` _string_ | _Optional._ The type of transition that led to the screen being viewed. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_mobile_screen_view_1 screen_view_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'screen_view'

and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Screen entity

Some trackers can automatically attach a `screen` entity to all events after a screen view is tracked. This entity contains information about the current, or last seen, screen.

### `screen`

**Type:** Schema

Schema for a context that represents information pertaining to the current screen being viewed when an event occurs.

**Properties:**

| Property                     | Description                                                            |
| ---------------------------- | ---------------------------------------------------------------------- |
| `name` _string_              | _Required._ The name of the screen viewed.                             |
| `type` _string_              | _Optional._ The type of screen that was viewed e.g feed / carousel.    |
| `id` _string_                | _Required._ An ID from the associated screenview event.                |
| `viewController` _string_    | _Optional._ iOS specific: class name of the view controller.           |
| `topViewController` _string_ | _Optional._ iOS specific: class name of the top level view controller. |
| `activity` _string_          | _Optional._ Android specific: name of activity.                        |
| `fragment` _string_          | _Optional._ Android specific: name of fragment.                        |

---

# Page element visibility and interaction tracking
> Track page element visibility, button clicks, and form interactions to understand how users engage with specific components on your web pages.
> Source: https://docs.snowplow.io/docs/events/ootb-data/page-elements/

Page element tracking captures user interactions with specific elements on your web pages, including buttons, forms, and any other visible components.

Use page element tracking to:

- Measure button click engagement across your site
- Track form interactions including focus, changes, and submissions
- Understand which page sections users actually see
- Build funnel analysis based on element visibility and interactions
- Measure content consumption and scroll depth

## Button clicks

Button click tracking automatically captures clicks on `<button>` and `<input type="button">` elements.

This table shows the support for button click tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                               |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ----------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/button-click/)           | ✅         | 3.18.0        | ✅             | Requires button click plugin        |
| iOS                                                                       | ❌         |               |               |                                     |
| Android                                                                   | ❌         |               |               |                                     |
| React Native                                                              | ❌         |               |               |                                     |
| Flutter                                                                   | ❌         |               |               |                                     |
| Roku                                                                      | ❌         |               |               |                                     |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v4            | ✅             | Integrates with button click plugin |

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes button clicks.

### `button_click`

**Type:** Event

Schema for a button click event

**Properties:**

| Property          | Description                                                     |
| ----------------- | --------------------------------------------------------------- |
| `label` _string_  | _Required._ The text on the button, or a user-provided override |
| `id` _string_     | _Optional._ The ID of the button                                |
| `classes` _array_ | _Optional._ The classes of the button                           |
| `name` _string_   | _Optional._ The name attribute of the button                    |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_button_click_1 button_click_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'button_click'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

## Form interactions

Form tracking captures three types of events: when a user focuses on a form field, when a field value changes, and when a form is submitted.

This table shows the support for form tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version                       | Auto-tracking | Notes                                                                                             |
| ------------------------------------------------------------------------- | --------- | ----------------------------------- | ------------- | ------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/form-tracking/)          | ✅         | 2.1.0 (directly), 3.0.0 (as plugin) | ✅             | Requires form tracking plugin                                                                     |
| iOS                                                                       | ❌         |                                     |               |                                                                                                   |
| Android                                                                   | ❌         |                                     |               |                                                                                                   |
| React Native                                                              | ❌         |                                     |               |                                                                                                   |
| Flutter                                                                   | ❌         |                                     |               |                                                                                                   |
| Roku                                                                      | ❌         |                                     |               |                                                                                                   |
| Python                                                                    | ❌         |                                     |               | The Python tracker has deprecated form tracking APIs; use custom events with form schemas instead |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3                                  | ✅             | Integrates with form tracking plugin                                                              |

### Focus form event

Triggered when a user focuses on a form field.

### `focus_form`

**Type:** Event

Schema for a form field focus event

**Properties:**

| Property                 | Description                                                                            |
| ------------------------ | -------------------------------------------------------------------------------------- |
| `formId` _string_        | _Required._ The ID of the form                                                         |
| `elementId` _string_     | _Optional._ The ID of the element                                                      |
| `nodeName` _string_      | _Required._ The node name of the element Must be one of: `INPUT`, `TEXTAREA`, `SELECT` |
| `elementType` _string_   | _Optional._ The type of the element                                                    |
| `elementClasses` _array_ | _Required._ The classes of the element                                                 |
| `value` _string_         | _Optional._ The value of the element                                                   |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_focus_form_1 focus_form_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'focus_form'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Change form event

Triggered when a user changes the value of a form field.

### `change_form`

**Type:** Event

Schema for a form field change event

**Properties:**

| Property                 | Description                                                                            |
| ------------------------ | -------------------------------------------------------------------------------------- |
| `formId` _string_        | _Required._ The ID of the form                                                         |
| `elementId` _string_     | _Optional._ The ID of the element                                                      |
| `nodeName` _string_      | _Required._ The node name of the element Must be one of: `INPUT`, `TEXTAREA`, `SELECT` |
| `elementType` _string_   | _Optional._ The type of the element                                                    |
| `elementClasses` _array_ | _Required._ The classes of the element                                                 |
| `value` _string_         | _Required._ The value of the element                                                   |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_change_form_1 change_form_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'change_form'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Submit form event

Triggered when a user submits a form. Contains data about all form fields.

### `submit_form`

**Type:** Event

Schema for a form submission event

**Properties:**

| Property              | Description                                    |
| --------------------- | ---------------------------------------------- |
| `formId` _string_     | _Required._ The ID of the form                 |
| `formClasses` _array_ | _Required._ The classes of the form            |
| `elements` _array_    | _Required._ The form elements and their values |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_submit_form_1 submit_form_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'submit_form'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

## Page element visibility and lifecycle

Element visibility tracking enables declarative tracking of page elements as they are created, become visible, leave view, or are removed from the page. It's only available for web.

This table shows the support for page element visibility tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                             | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------- | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/element-tracking/) | ✅         | 4.6.0         | ✅             |
| iOS                                                                 | ❌         |               |               |
| Android                                                             | ❌         |               |               |
| React Native                                                        | ❌         |               |               |
| Flutter                                                             | ❌         |               |               |
| Roku                                                                | ❌         |               |               |
| Google Tag Manager                                                  | ❌         |               |               |

### Events

The element tracking plugin generates four event types:

- `create_element`: when a matching element is added to the page
- `expose_element`: when a matching element scrolls into view
- `obscure_element`: when a matching element scrolls out of view
- `destroy_element`: when a matching element is removed from the page

All of these events have a single property, `element_name`. The plugin adds an `element` entity with information about the tracked element.

You can also configure the tracker to attach `element_statistics`, `element_content`, or `component_parents` entities.

#### Create element event

Triggered when an element matching a named configuration is added to the page or detected on page load.

### `create_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as existing in or being added to a document.

**Properties:**

| Property                | Description                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was created. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_create_element_1 create_element_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'create_element'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

#### Expose element event

Triggered when a tracked element scrolls into the viewport and becomes visible to the user.

### `expose_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as intersecting with the viewport, becoming visible to the user.

**Properties:**

| Property                | Description                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was exposed. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_expose_element_1 expose_element_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'expose_element'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

#### Obscure element event

Triggered when a tracked element scrolls out of the viewport or becomes hidden from the user.

### `obscure_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as becoming hidden from a user or moving out of the viewport.

**Properties:**

| Property                | Description                                                                                                                                   |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was obscured. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_obscure_element_1 obscure_element_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'obscure_element'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

#### Destroy element event

Triggered when a tracked element is removed from the page.

### `destroy_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as being removed from a document.

**Properties:**

| Property                | Description                                                                                                                                    |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was destroyed. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_destroy_element_1 destroy_element_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'destroy_element'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Element entity

The `element` entity is attached to all element tracking events and contains information about the tracked element's position, dimensions, and attributes.

### `element`

**Type:** Schema

Schema for element context

**Properties:**

| Property                         | Description                                                       |
| -------------------------------- | ----------------------------------------------------------------- |
| `element_name` _string_          | _Required._ The name of the element                               |
| `width` _number_                 | _Optional._ The width of the element                              |
| `height` _number_                | _Optional._ The height of the element                             |
| `position_x` _number_            | _Optional._ The x position of the element in the viewport         |
| `position_y` _number_            | _Optional._ The y position of the element in the viewport         |
| `doc_position_x` _number_        | _Optional._ The x position of the element in the document         |
| `doc_position_y` _number_        | _Optional._ The y position of the element in the document         |
| `element_index` _integer_        | _Optional._ The index of this element among all matching elements |
| `element_matches` _integer_      | _Optional._ The total number of elements matching the rule        |
| `originating_page_view` _string_ | _Optional._ The page view ID when the element was first observed  |
| `attributes` _array_             | _Optional._ Custom attributes extracted from the element          |

### Element statistics entity

The `element_statistics` entity contains information about element visibility and scroll depth.

### `element_statistics`

**Type:** Schema

Schema for element statistics

**Properties:**

| Property                          | Description |
| --------------------------------- | ----------- |
| `element_name` _string_           | _Required._ |
| `element_index` _integer_         | _Optional._ |
| `element_matches` _integer_       | _Optional._ |
| `current_state` _string_          | _Optional._ |
| `min_size` _string_               | _Optional._ |
| `current_size` _string_           | _Optional._ |
| `max_size` _string_               | _Optional._ |
| `y_depth_ratio` _number_          | _Optional._ |
| `max_y_depth_ratio` _number_      | _Optional._ |
| `max_y_depth` _string_            | _Optional._ |
| `element_age_ms` _integer_        | _Optional._ |
| `times_in_view` _integer_         | _Optional._ |
| `total_time_visible_ms` _integer_ | _Optional._ |

### Element content entity

The `element_content` entity captures data extracted from a tracked element, including its position within a parent component hierarchy.

### `element_content`

**Type:** Schema

Entity describing the content of an element matching a named configuration.

**Properties:**

| Property                  | Description                                                                                                                                                                               |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `parent_name` _string_    | _Required._ The name of the configuration for the element/component that contains the element described by this entity.                                                                   |
| `parent_index` _integer_  | _Required._ The index of this element's parent element/component's within it's parent's other matches for it's element configuration.                                                     |
| `element_name` _string_   | _Required._ The name of the configuration for the element/component that contains this data. This will usually match the corresponding element\_name in an event payload or other entity. |
| `element_index` _integer_ | _Required._ The position of this element's within it's parent's other matches for the element's configuration.                                                                            |
| `attributes` _array_      | _Optional._ Results of configured contents descriptions found on this element.                                                                                                            |

### Component parents entity

The `component_parents` entity records the hierarchy of parent components that contain a tracked element.

### `component_parents`

**Type:** Schema

Entity describing the list of components that were found to contain the element with the named configuration.

**Properties:**

| Property                 | Description                                                                                                                                                                      |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_  | _Optional._ Name of the element that this entity relates to, if any. If not provided, may apply to a subject of some other event, such as Link, Button, or Form Tracking events. |
| `component_list` _array_ | _Required._ List of component names that were detected as containing the element that is the subject of this event.                                                              |

---

# Site search tracking
> Track internal site search queries with search terms, filters, and result counts to analyze user search behavior and content discovery.
> Source: https://docs.snowplow.io/docs/events/ootb-data/site-search/

Site search tracking captures how users search within your website. This data helps you understand what content users are looking for, whether they find it, and how search influences their journey through your site.

Use site search tracking to:

- Identify popular search terms and content gaps
- Measure search success rates based on result counts
- Analyze how search filters affect user behavior
- Optimize your internal search functionality

This table shows the support for site search tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                         |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ----------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/site-search/)            | ✅         | 3.0.0         | ❌             | Requires site tracking plugin |
| iOS                                                                       | ❌         |               |               |                               |
| Android                                                                   | ❌         |               |               |                               |
| React Native                                                              | ❌         |               |               |                               |
| Flutter                                                                   | ❌         |               |               |                               |
| Roku                                                                      | ❌         |               |               |                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3            | ❌             |                               |

The site search event captures the search terms, any filters applied, and the number of results returned.

### `site_search`

**Type:** Event

Schema for a site search event

**Properties:**

| Property                 | Description                                   |
| ------------------------ | --------------------------------------------- |
| `terms` _array_          | _Required._ The search terms                  |
| `filters` _object_       | _Optional._ The search filters                |
| `totalResults` _integer_ | _Optional._ The total number of results       |
| `pageResults` _integer_  | _Optional._ The number of results on the page |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_site_search_1 site_search_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'site_search'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Social media interaction tracking
> Track user interactions with social media widgets including likes, shares, and retweets across Facebook, Twitter, and other platforms.
> Source: https://docs.snowplow.io/docs/events/ootb-data/social-media/

Social media interaction tracking captures how users engage with social widgets embedded on your website, such as like buttons, share buttons, and tweet buttons.

Use social interaction tracking to:

- Measure how often content is shared on social platforms
- Identify which social networks drive the most engagement
- Understand which content resonates with users enough to share
- Track the performance of social sharing features

This table shows the support for social interaction tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                         |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ----------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/social-media/)           | ✅         | 3.0.0         | ❌             | Requires site tracking plugin |
| iOS                                                                       | ❌         |               |               |                               |
| Android                                                                   | ❌         |               |               |                               |
| React Native                                                              | ❌         |               |               |                               |
| Flutter                                                                   | ❌         |               |               |                               |
| Roku                                                                      | ❌         |               |               |                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3            | ❌             |                               |

The social interaction event captures the social network, the action performed, and optionally the target content.

### `social_interaction`

**Type:** Event

Schema for a social interaction event

**Properties:**

| Property           | Description                                                               |
| ------------------ | ------------------------------------------------------------------------- |
| `action` _string_  | _Required._ The social action performed, e.g. like, retweet, share        |
| `network` _string_ | _Required._ The social network, e.g. facebook, twitter                    |
| `target` _string_  | _Optional._ The object of the social action, e.g. a page ID or product ID |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_social_interaction_1 social_interaction_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'social_interaction'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Third-party sources
> Capture data from third-party tools including Optimizely experiments, Google Analytics cookies, and Kantar Focal Meter for unified analytics.
> Source: https://docs.snowplow.io/docs/events/ootb-data/third-party-sources/

Snowplow provides tracking integrations that let you capture data from third-party tools deployed on your web and mobile properties.

## Optimizely X

The Optimizely X web plugin captures experiment and variation data from [Optimizely](https://www.optimizely.com/) A/B tests. The tracker will add it to all tracked events as an entity.

This table shows the support for Optimizely X tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                       | Supported | Since version | Auto-tracking | Notes                        |
| ------------------------------------------------------------- | --------- | ------------- | ------------- | ---------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/optimizely/) | ✅         | 3.0.0         | ✅             | Requires Optimizely X plugin |
| iOS                                                           | ❌         |               |               |                              |
| Android                                                       | ❌         |               |               |                              |
| React Native                                                  | ❌         |               |               |                              |
| Flutter                                                       | ❌         |               |               |                              |
| Roku                                                          | ❌         |               |               |                              |
| Google Tag Manager                                            | ❌         |               |               |                              |

### `summary`

**Type:** Schema

Schema for Optimizely X experiment context

**Properties:**

| Property                 | Description                                               |
| ------------------------ | --------------------------------------------------------- |
| `experimentId` _integer_ | _Optional._ The Optimizely experiment ID                  |
| `variationName` _string_ | _Optional._ The variation name (requires privacy setting) |
| `variationId` _integer_  | _Optional._ The variation ID                              |
| `isActive` _boolean_     | _Optional._ Whether the experiment is active              |

## Google Analytics cookies

The GA cookies web plugin captures Google Analytics cookie values (both GA4 and Universal Analytics) and attaches them to all events.

This table shows the support for GA cookie tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                       | Supported | Since version | Auto-tracking | Notes                      |
| ------------------------------------------------------------- | --------- | ------------- | ------------- | -------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/ga-cookies/) | ✅         | 3.0.0         | ✅             | Requires GA cookies plugin |
| iOS                                                           | ❌         |               |               |                            |
| Android                                                       | ❌         |               |               |                            |
| React Native                                                  | ❌         |               |               |                            |
| Flutter                                                       | ❌         |               |               |                            |
| Roku                                                          | ❌         |               |               |                            |
| Google Tag Manager                                            | ❌         |               |               |                            |

The entity schema depends on whether the tracker finds GA4 or Universal Analytics cookies.

GA4 schema:

### `cookies`

**Type:** Schema

Schema for GA4 cookies context

**Properties:**

| Property                  | Description                                         |
| ------------------------- | --------------------------------------------------- |
| `_ga` _string_            | _Optional._ The \_ga cookie value                   |
| `cookie_prefix` _string_  | _Optional._ The cookie prefix if configured         |
| `session_cookies` _array_ | _Optional._ Session cookie values by measurement ID |

Universal Analytics schema:

### `cookies`

**Type:** Schema

Schema for Universal Analytics cookies context

**Properties:**

| Property          | Description                       |
| ----------------- | --------------------------------- |
| `_ga` _string_    | _Optional._ The \_ga cookie value |
| `__utma` _string_ | _Optional._ Classic GA cookie     |
| `__utmb` _string_ | _Optional._ Classic GA cookie     |
| `__utmc` _string_ | _Optional._ Classic GA cookie     |
| `__utmv` _string_ | _Optional._ Classic GA cookie     |
| `__utmz` _string_ | _Optional._ Classic GA cookie     |

## Kantar Focal Meter

Some Snowplow trackers integrate with [Kantar Focal Meter](https://www.virtualmeter.co.uk/focalmeter), a router-based audience measurement system. The plugin sends user IDs to Focal Meter endpoints to enable content audience measurement.

This integration sends data to Kantar's endpoint, rather than attaching an entity to events.

This table shows the support for Kantar Focal Meter tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                       | Supported | Since version | Auto-tracking | Notes                       |
| ------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/focalmeter/) | ✅         | 3.16.0        | ✅             | Requires Focal Meter plugin |
| [iOS](/docs/sources/mobile-trackers/plugins/focal-meter/)     | ✅         | 5.6.0         | ❌             |                             |
| [Android](/docs/sources/mobile-trackers/plugins/focal-meter/) | ✅         | 5.6.0         | ❌             |                             |
| React Native                                                  | ❌         |               |               |                             |
| Flutter                                                       | ❌         |               |               |                             |
| Roku                                                          | ❌         |               |               |                             |
| Google Tag Manager                                            | ❌         |               |               |                             |

## Privacy Sandbox (deprecated)

> **Note:** Privacy Sandbox has been deprecated by Google. This plugin remains available but the underlying browser API may no longer be supported.

The Privacy Sandbox plugin captures data from the [Topics API](https://developer.chrome.com/docs/privacy-sandbox/topics/overview/), which was designed to provide privacy-preserving interest-based advertising signals.

This table shows the support for Privacy Sandbox tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                            | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------ | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/privacy-sandbox/) | ✅         | 3.14.0        | ✅             |
| iOS                                                                | ❌         |               |               |
| Android                                                            | ❌         |               |               |
| React Native                                                       | ❌         |               |               |
| Flutter                                                            | ❌         |               |               |
| Roku                                                               | ❌         |               |               |
| Google Tag Manager                                                 | ❌         |               |               |

### `topics`

**Type:** Schema

Schema for Privacy Sandbox Topics context

**Properties:**

| Property         | Description                                     |
| ---------------- | ----------------------------------------------- |
| `topics` _array_ | _Optional._ List of topics for the current user |

---

# User and session identifiers
> Track user and session identifiers including domain_userid, network_userid, session IDs, and custom business user IDs.
> Source: https://docs.snowplow.io/docs/events/ootb-data/user-and-session-identification/

Snowplow tracks a number of different identifiers to help you understand user behavior across sessions and devices. Read more about these in the [identifiers overview](/docs/events/identifiers/) page.

The main user and session identifiers are tracked as:

- Atomic event properties
- Session entity properties

Not all trackers have session tracking capabilities.

## User atomic event properties

These [properties](/docs/fundamentals/canonical-event/#user-fields) can be assigned using most Snowplow trackers: `user_id`, `domain_userid`, `network_userid`, `domain_sessionid`, `domain_sessionidx`, and `user_ipaddress`.

Some trackers set certain user properties automatically, while others require you to set them manually using the tracker's API.

If you don't manually set an IP address or `network_userid`, the event [Collector](/docs/pipeline/collector/) will automatically add them to the event based on the request HTTP headers and the Collector cookie. You can turn this feature off using [server-side anonymization](/docs/events/anonymous-tracking/).

Values set in your tracking code will take precedence over values added by the Collector or enrichments.

| Tracker                                                                      | Supported | Since version | Notes                                                                                                                                                                                                    |
| ---------------------------------------------------------------------------- | --------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#business-user-id)          | ✅         | 0.1.0         | Set `user_id` only. Tracker sets the `domain_userid`, `network_userid`, `domain_sessionid`, and `domain_sessionidx` automatically from [cookies](/docs/sources/web-trackers/cookies-and-local-storage/). |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.5.0         | Set all properties except `domain_sessionid` and `domain_sessionidx`.                                                                                                                                    |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         | Set all properties except `domain_sessionid` and `domain_sessionidx`.                                                                                                                                    |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         | Set `user_id` only on web. Set `user_id`, `domain_userid`, `network_userid`, and `user_ipaddress` on mobile.                                                                                             |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.2.0         | Set `user_id` and `network_userid` only. You can automatically populate `domain_userid`, `domain_sessionid`, and `domain_sessionidx` by configuring `sessionAsCookies`.                                  |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         | Set all properties.                                                                                                                                                                                      |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 1.0.0         | Set all properties.                                                                                                                                                                                      |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.10.0        | Set all properties except `domain_sessionidx`.                                                                                                                                                           |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.13.0        | Set all properties.                                                                                                                                                                                      |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.6.0         | Set all properties except `domain_sessionid` and `domain_sessionidx`.                                                                                                                                    |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         | Set all properties.                                                                                                                                                                                      |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         | Set all properties except `domain_sessionidx`.                                                                                                                                                           |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         | Set `user_id` and `user_ipaddress` only.                                                                                                                                                                 |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                   | ✅         | 0.1.0         | Set `user_id` only.                                                                                                                                                                                      |
| Google Tag Manager                                                           | ❌         |               | Set using custom configuration.                                                                                                                                                                          |

## Session timeouts

A session is a period of time in which a user interacts with your application. A new session starts the first time an event is tracked after the previous session's timeout has elapsed.

There's no maximum session duration, but sessions do have a configurable inactivity timeout. If there's no user activity for the specified time, the session ends. A new one will start when the user returns.

For most trackers, the timeout is based on the gap between the last tracked event and the current event. Mobile trackers have two session timeout settings, allowing you to specify different timeouts if the app is in the foreground or background.

| Tracker                                                                              | Foreground timeout, min | Background timeout, min | Notes                                                                                                         |
| ------------------------------------------------------------------------------------ | ----------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/session/)                           | 30                      | -                       | Page pings from [activity tracking](/docs/events/ootb-data/page-activity-tracking/) will keep a session alive |
| [iOS](/docs/sources/mobile-trackers/tracking-events/session-tracking/)               | 30                      | 30                      |                                                                                                               |
| [Android](/docs/sources/mobile-trackers/tracking-events/session-tracking/)           | 30                      | 30                      |                                                                                                               |
| [React Native](/docs/sources/react-native-tracker/tracking-events/session-tracking/) | 30                      | 30                      |                                                                                                               |
| [Flutter](/docs/sources/flutter-tracker/sessions-and-data-model/)                    | 30                      | 30                      |                                                                                                               |
| [Roku](/docs/sources/roku-tracker/adding-data/)                                      | 30                      | -                       |                                                                                                               |
| [.NET](/docs/sources/net-tracker/setup/)                                             | 10                      | 5                       | Uses a timer so it can expire a session independently of a new tracked event                                  |
| [C++](/docs/sources/c-tracker/client-sessions/)                                      | 30                      | 30                      |                                                                                                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)            | 30                      | -                       |                                                                                                               |

## Session entity

Some trackers can track an entity containing information about the current session.

This table shows the support for `client_session` entity tracking across the main [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                              | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------------------------ | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/session/)                           | ✅         | 3.5.0         | ✅             |
| [iOS](/docs/sources/mobile-trackers/tracking-events/session-tracking/)               | ✅         | 0.5.0         | ✅             |
| [Android](/docs/sources/mobile-trackers/tracking-events/session-tracking/)           | ✅         | 0.5.0         | ✅             |
| [React Native](/docs/sources/react-native-tracker/tracking-events/session-tracking/) | ✅         | 1.0.0         | ✅             |
| [Flutter](/docs/sources/flutter-tracker/sessions-and-data-model/)                    | ✅         | 0.2.0         | ✅             |
| [Roku](/docs/sources/roku-tracker/adding-data/)                                      | ✅         | 0.3.0         | ✅             |
| Node.js                                                                              | ❌         |               |               |
| Golang                                                                               | ❌         |               |               |
| [.NET](/docs/sources/net-tracker/setup/)                                             | ✅         | 1.0.0         | ✅             |
| Java                                                                                 | ❌         |               |               |
| Python                                                                               | ❌         |               |               |
| Scala                                                                                | ❌         |               |               |
| Ruby                                                                                 | ❌         |               |               |
| Rust                                                                                 | ❌         |               |               |
| PHP                                                                                  | ❌         |               |               |
| [C++](/docs/sources/c-tracker/client-sessions/)                                      | ✅         | 1.0.0         | ✅             |
| Unity                                                                                | ❌         |               |               |
| Lua                                                                                  | ❌         |               |               |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)            | ✅         | v4            | ✅             |

The `client_session` entity contains data such as the session identifier and session index, as well as information about the previous session to help analysis.

The `userId` property is the device ID.

### `client_session`

**Type:** Event

Schema for a client-generated user session

**Properties:**

| Property                     | Description                                                                                                                                                         |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `userId` _string_            | _Required._ An identifier for the user of the session. It is set on app install.                                                                                    |
| `sessionId` _string_         | _Required._ An identifier for the session                                                                                                                           |
| `sessionIndex` _integer_     | _Required._ The index of the current session for this user                                                                                                          |
| `eventIndex` _null_          | _Optional._ Optional index of the current event in the session                                                                                                      |
| `previousSessionId` _null_   | _Required._ The previous session identifier for this user                                                                                                           |
| `storageMechanism` _string_  | _Required._ The mechanism that the session information has been stored on the device Must be one of: `SQLITE`, `COOKIE_1`, `COOKIE_3`, `LOCAL_STORAGE`, `FLASH_LSO` |
| `firstEventId` _null_        | _Optional._ The optional identifier of the first event for this session                                                                                             |
| `firstEventTimestamp` _null_ | _Optional._ Optional date-time timestamp of when the first event in the session was tracked                                                                         |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_client_session_1 client_session_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'client_session'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# visionOS and SwiftUI
> Track window groups and immersive spaces in visionOS apps on Apple Vision Pro using SwiftUI context entities and events.
> Source: https://docs.snowplow.io/docs/events/ootb-data/visionos-swiftui/

Use the window group and immersive space entities and events to understand user behavior within visionOS apps on Apple's Vision Pro headset.

Track the opening and dismissing of SwiftUI window groups using `OpenWindowEvent` and `DismissWindowEvent`. These events can be used in any SwiftUI app, not just visionOS. The event data is sent as a window group entity attached to these events.

Use the `OpenImmersiveSpaceEvent` and `DismissImmersiveSpaceEvent` to automatically add an immersive space entity to all events occurring within an immersive space. The entity will identify the immersive space in which the events occurred. This feature is enabled by default for the iOS tracker.

This table shows the support for VisionOS tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). Snowplow currently only supports visionOS apps via SwiftUI, not Unity.

| Tracker                                                        | Supported | Since version | Auto-tracking | Notes                            |
| -------------------------------------------------------------- | --------- | ------------- | ------------- | -------------------------------- |
| Web                                                            | ❌         |               |               |                                  |
| [iOS](/docs/sources/mobile-trackers/tracking-events/visionos/) | ✅         | 6.0.0         | ✅/❌           | Entities are automatically added |
| Android                                                        | ❌         |               |               |                                  |
| React Native                                                   | ❌         |               |               |                                  |
| Flutter                                                        | ❌         |               |               |                                  |
| Roku                                                           | ❌         |               |               |                                  |
| Unity                                                          | ❌         |               |               |                                  |
| Google Tag Manager                                             | ❌         |               |               |                                  |

## Available events

The following events track user interactions with SwiftUI windows and visionOS immersive spaces. These events have no properties themselves; the contextual data is captured in the attached entities.

### Open window

Tracks when a user opens a SwiftUI window or window group.

### `open_window`

**Type:** Event

Schema for an event for opening a SwiftUI window or window group.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_apple_swiftui_open_window_1 open_window_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'open_window'

and events.event_vendor = 'com.apple.swiftui'
```

### Dismiss window

Tracks when a user closes or dismisses a SwiftUI window or window group.

### `dismiss_window`

**Type:** Event

Schema for an event for dismissing a SwiftUI window or window group.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_apple_swiftui_dismiss_window_1 dismiss_window_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'dismiss_window'

and events.event_vendor = 'com.apple.swiftui'
```

### Open immersive space

Tracks when a user enters an immersive space in a visionOS app on Apple Vision Pro.

### `open_immersive_space`

**Type:** Event

Schema for an event for opening a visionOS immersive space.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_apple_swiftui_open_immersive_space_1 open_immersive_space_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'open_immersive_space'

and events.event_vendor = 'com.apple.swiftui'
```

### Dismiss immersive space

Tracks when a user exits an immersive space in a visionOS app.

### `dismiss_immersive_space`

**Type:** Event

Schema for an event for dismissing a visionOS immersive space.

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_apple_swiftui_dismiss_immersive_space_1 dismiss_immersive_space_1

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'dismiss_immersive_space'

and events.event_vendor = 'com.apple.swiftui'
```

## Available entities

These entities provide context about the SwiftUI windows and visionOS immersive spaces where events occur.

### Window group

Describes a SwiftUI window group, including its identifier, title, and window style. This entity is attached to open and dismiss window events.

### `window_group`

**Type:** Schema

Schema for a window group entity, representing the SwiftUI window group that the event occurs in.

**Properties:**

| Property                | Description                                                                                                                                                               |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `window_id` _string_    | _Optional._ UUID for the current window within the group.                                                                                                                 |
| `id` _string_           | _Required._ A string that uniquely identifies the window group. Identifiers must be unique among the window groups in your app.                                           |
| `title_key` _string_    | _Optional._ A localized string key to use for the window's title in system menus and in the window's title bar. Provide a title that describes the purpose of the window. |
| `window_style` _string_ | _Optional._ A specification for the appearance and interaction of a window. Must be one of: `automatic`, `hiddenTitleBar`, `plain`, `titleBar`, `volumetric`,``           |

### Immersive space

Describes a visionOS immersive space, including its immersion style and upper limb visibility settings. This entity is automatically attached to all events that occur within an immersive space.

### `immersive_space`

**Type:** Schema

Schema for an immersive space entity, representing the VisionOS immersive space that the event occurs in.

**Properties:**

| Property                         | Description                                                                                                                                                  |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `view_id` _string_               | _Optional._ UUID for the view of the immersive space.                                                                                                        |
| `id` _string_                    | _Required._ The identifier of the immersive space to present.                                                                                                |
| `immersion_style` _string_       | _Optional._ The style of an immersive space. Must be one of: `automatic`, `full`, `mixed`, `progressive`,``                                                  |
| `upper_limb_visibility` _string_ | _Optional._ Preferred visibility of the user's upper limbs, while an immersive space scene is presented. Must be one of: `automatic`, `visible`, `hidden`,`` |

---

# Timestamps
> Understanding Snowplow event timestamps from creation through the pipeline, including derived timestamp calculation and usage recommendations.
> Source: https://docs.snowplow.io/docs/events/timestamps/

Snowplow events have multiple timestamps that are added as the payload moves through the pipeline. The set of timestamps is designed to account for devices with incorrectly set clocks, or delays in event sending due to network outages. All timestamps are converted to UTC for consistency across events.

The timestamps are:

| Timestamp             | Added by                                                    | Description                                                                                                                                                    | In all events |
| --------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `dvce_created_tstamp` | [Tracker](/docs/sources/)                                   | The device's timestamp when the event was created - not necessarily the correct time                                                                           | ✅             |
| `dvce_sent_tstamp`    | [Tracker](/docs/sources/)                                   | The device's timestamp when the event was successfully sent to the Collector endpoint - not necessarily the correct time                                       | ✅             |
| `true_tstamp`         | You                                                         | Exact timestamp, defined within your tracking code                                                                                                             | ❌             |
| `collector_tstamp`    | [Collector](/docs/pipeline/collector/)                      | When the Collector received the event payload                                                                                                                  | ✅             |
| `derived_tstamp`      | [Enrich](/docs/api-reference/enrichment-components/)        | Calculated from other timestamps, or the same as `true_tstamp`                                                                                                 | ✅             |
| `etl_tstamp`          | [Enrich](/docs/api-reference/enrichment-components/)        | The time at which Enrich started processing the event                                                                                                          | ✅             |
| `refr_device_tstamp`  | [Enrich](/docs/api-reference/enrichment-components/)        | Timestamp extracted from the [cross-domain navigation query string](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/), if present | ❌             |
| `load_tstamp`         | [Loader](/docs/destinations/warehouses-lakes/) or warehouse | Timestamp when the event was loaded into the warehouse                                                                                                         | ✅             |

The `load_tstamp` is added either by the Loader or by the warehouse at the point of loading, depending on the Loader/warehouse. Use this timestamp for incremental processing.

## Derived timestamp

The `derived_tstamp` corrects for inaccurate timestamps and delays.

It's calculated as `collector_tstamp` minus the difference between the `dvce_sent_tstamp` and the `dvce_created_tstamp`. Alternatively, if `true_tstamp` is available, `derived_tstamp` is the same as `true_tstamp`.

The calculation has two assumptions:

- We assume that, although `dvce_created_tstamp` and `dvce_sent_tstamp` can both be inaccurate, they're inaccurate in precisely the same way. If the device clock is 15 minutes fast at event creation, then it remains 15 minutes fast at event sending, whenever that might be.
- We assume that the time taken for an event to get from the device to the Collector is neglible, i.e. the lag between `dvce_sent_tstamp` and `collector_tstamp` is 0 seconds.

## What timestamp to use?

The choice of timestamp for downstream analysis depends on how you're using the data.

For reporting, we recommend using the `derived_tstamp`. Our data models use `derived_tstamp` as the default `start_tstamp` for creating derived tables.

To filter the `atomic.events` table directly, use the timestamp it's partitioned on. For example, if your warehouse is partitioned by `load_tstamp`, then use clauses like `WHERE load_tstamp > ?` in your analysis.

The `refr_device_tstamp` timestamp can help you understand cross-domain navigation by your users.

All of the timestamps can also be used for custom use cases, or in debugging, to closely follow how the events are created and move through the pipeline.

---

# Snowplow atomic event properties
> A summary of the Snowplow events table and its standard fields
> Source: https://docs.snowplow.io/docs/fundamentals/canonical-event/

This page provides a reference for the standard fields that are found in all Snowplow events. These fields define the Snowplow tracker protocol, and are often called "atomic properties" or "atomic fields" in reference to the [Snowplow `atomic.events` table](/docs/fundamentals/warehouse-tables/). They're also sometimes called "canonical fields".

Fields that are populated during tracking have two names associated with them:

- **Payload property**: a short payload name, to reduce the size of the HTTP request
- **Field name**: a more descriptive column or field name

The field **type** refers to the data type in the enriched event data. For some fields, the type is different for the raw tracker payload and the enriched event. Again, this is to reduce the size of the HTTP request.

Different event fields are populated by different applications, such as tracker SDKs or [enrichments](/docs/pipeline/enrichments/). The **source** of data for each field is indicated in the tables below.

Values you provide in your tracking code have precedence over pipeline-derived values. For example, if you [set `useragent`](/docs/events/ootb-data/device-and-browser/#user-agent) in your tracking code, the [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) will use that value, rather than overriding it with the value from the `User-Agent` HTTP header.

## Validation

During enrichment, atomic property values are validated against the [atomic schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0). If a required field is missing or invalid, for example the wrong type or length, [Enrich](/docs/api-reference/enrichment-components/) will process the event as a [failed event](/docs/fundamentals/failed-events/). The **reqd?** values in these tables specify whether a field is required for a Snowplow event.

The tables show the default maximum lengths for string fields. You can configure different maximum lengths for specific fields by setting `atomicFieldsLimits` in the [Enrich configuration](/docs/api-reference/enrichment-components/configuration-reference/). These can be more permissive than the limits in the atomic schema. In fact, the `page_url`, `page_referrer`, and `mkt_clickid` fields already have larger maximum lengths in the default Enrich configuration than in the atomic schema.

Some identifiers require UUID strings. Snowplow components use UUIDv4, but the pipeline will accept any valid UUID, regardless of version.

Any payload that conforms to this protocol is a valid Snowplow event payload, whether it's sent by a Snowplow tracker SDK, a webhook, or a custom application. If you want to get into the details, check out these [example HTTP requests](/docs/events/http-requests/). In total, the tracker protocol defines 131 fields, of which 89 are in use by Snowplow applications.

## Common fields

These fields are common to all Snowplow events, regardless of platform or event type.

### Event fields

The `event_id` is the unique identifier (UUID) for the table row. This should be generated by trackers, but if missing will be generated by the enrichment process.

| Payload property | Field name          | Type                     | Description                                           | Reqd? | Example                                | Source                                                                                                         | Web | Mobile |
| ---------------- | ------------------- | ------------------------ | ----------------------------------------------------- | ----- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------- | --- | ------ |
| `e`              | `event`             | `string`, max length 128 | The type of event recorded                            | Yes   | `page_view`                            | Tracking                                                                                                       | ✅   | ✅      |
| `eid`            | `event_id`          | UUID `string`            | A UUID for each event                                 | Yes   | `c6ef3124-b53a-4b13-a233-0088f79dcbcb` | Tracking (or enrichment if empty)                                                                              | ✅   | ✅      |
|                  | `event_fingerprint` | `string`, max length 128 | Hash client-set event fields, used to de-dupe records | No    | `AADCE520E20C2899F4CED228A79A3083`     | [Event fingerprint enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/) | ✅   | ✅      |

This table shows the possible values for the `event` field:

| Event type              | Payload `e` value | Field `event` value                  |
| ----------------------- | ----------------- | ------------------------------------ |
| Self-describing event   | `ue`              | `unstruct`                           |
| Page view               | `pv`              | `page_view`                          |
| Page ping               | `pp`              | `page_ping`                          |
| Structured event        | `se`              | `struct`                             |
| Legacy ecommerce events | `tr` and `ti`     | `transaction` and `transaction_item` |

### User fields

Read more about tracking and using these fields in the [OOTB user and session data](/docs/events/ootb-data/user-and-session-identification/) and [identifiers](/docs/events/identifiers/) pages.

The `domain_userid` is regarded as the most reliable session based identifier for most use cases. It's treated as the primary `user_identifier` field in our data models that rely on sessionization, including the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) data model.

The `domain_sessionidx` is the number or index of the current user session. For example, an event occurring during a user's first session would have `domain_sessionidx` set to 1. The JavaScript tracker calculates this field by storing a visit count in a [first-party cookie](/docs/sources/web-trackers/cookies-and-local-storage/).

The equivalent values on mobile are tracked in a [session entity](/docs/events/ootb-data/user-and-session-identification/#session-entity).

The `network_userid` is set by a [Collector cookie](/docs/pipeline/collector/) by default. You can override it by setting a `network_userid` with your tracker.

The IP address is also added to the event by the Collector, if you didn't provide one in your tracking code. Snowplow supports both IPv4 and IPv6 addresses, as long as a domain is compatible with the IPv6 network.

| Payload property | Field name          | Type                     | Description                                                                    | Reqd? | Example                                | Source                                                                                                                   | Web | Mobile |
| ---------------- | ------------------- | ------------------------ | ------------------------------------------------------------------------------ | ----- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --- | ------ |
| `uid`            | `user_id`           | `string`, max length 255 | Unique identifier for user, set by the business using `setUserId`              | No    | `c94f860b-1266-4dad-ae57-3a36a414a521` | Tracking                                                                                                                 | ✅   | ✅      |
| `duid`           | `domain_userid`     | `string`, max length 128 | Unique identifier for a user, based on a first party cookie                    | No    | `4b0dfa75-9a8c-46a1-9691-01add9db4200` | Tracking                                                                                                                 | ✅   | ❌      |
| `tnuid`          | `network_userid`    | `string`, max length 128 | User ID set by Snowplow using server-set cookie                                | No    | `ecdff4d0-9175-40ac-a8bb-325c49733607` | Tracking or pipeline                                                                                                     | ✅   | ✅      |
| `sid`            | `domain_sessionid`  | UUID `string`            | Unique identifier (UUID) for this visit of this `domain_userid` to this domain | No    | `c6ef3124-b53a-4b13-a233-0088f79dcbcb` | Tracking                                                                                                                 | ✅   | ❌      |
| `vid`            | `domain_sessionidx` | `integer`                | Index of number of visits that this `domain_userid` has made to this domain    | No    | `3`                                    | Tracking                                                                                                                 | ✅   | ❌      |
| `ip`             | `user_ipaddress`    | `string`, max length 128 | User IP address, can be overwritten with the IP anonymization enrichment       | No    | `92.231.54.234`                        | Tracking or [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/) | ✅   | ✅      |

### Application fields

Use the application ID to distinguish events tracked from different applications or websites by the same Snowplow stack.

The platform ID is used to distinguish the same app running on different platforms, e.g. `iOS` vs `web`.

| Payload property | Field name     | Type                     | Description              | Reqd? | Example             | Source   | Web | Mobile |
| ---------------- | -------------- | ------------------------ | ------------------------ | ----- | ------------------- | -------- | --- | ------ |
| `aid`            | `app_id`       | `string`, max length 255 | Application ID           | Yes   | `snow-game-android` | Tracking | ✅   | ✅      |
| `p`              | `platform`     | `string`, max length 255 | Platform the app runs on | Yes   | `web`               | Tracking | ✅   | ✅      |
| `tna`            | `name_tracker` | `string`, max length 128 | Tracker namespace        | No    | `tracker_1`         | Tracking | ✅   | ✅      |

This table shows the possible values for the `platform` field:

| Platform                  | `platform` value |
| ------------------------- | ---------------- |
| Web, including mobile web | `web`            |
| Mobile, tablet            | `mob`            |
| Desktop, laptop           | `pc`             |
| Server-side application   | `srv`            |
| General application       | `app`            |
| Connected TV              | `tv`             |
| Games console             | `cnsl`           |
| Internet of Things        | `iot`            |
| Headset (e.g., AR, VR)    | `headset`        |

> **Info:** The tracker namespace parameter is used to distinguish between different trackers. The name can be any string that doesn't contain a colon or semicolon character. Tracker namespacing allows you to run multiple trackers, pinging to different collectors.

### Time and date fields

The `etl_tstamp` field records when the event was validated and enriched, not when it was loaded into the warehouse. The name is historical.

Most Snowplow trackers have built-in capability for tracking `os_timezone`. See the [geolocation and timezone tracking](/docs/events/ootb-data/geolocation/) page for tracker-specific configuration details.

| Payload property | Field name            | Type                     | Description                                                 | Reqd? | Example                   | Source             | Web | Mobile |
| ---------------- | --------------------- | ------------------------ | ----------------------------------------------------------- | ----- | ------------------------- | ------------------ | --- | ------ |
|                  | `collector_tstamp`    | `timestamp`              | Timestamp for the event recorded by the Collector           | Yes   | `2013-11-26 00:02:05.123` | Pipeline           | ✅   | ✅      |
| `dtm`            | `dvce_created_tstamp` | `timestamp`              | Timestamp for the event recorded on the client device       | No    | `2013-11-26 00:03:57.885` | Tracking           | ✅   | ✅      |
| `stm`            | `dvce_sent_tstamp`    | `timestamp`              | Timestamp when event occurred, as recorded by client device | No    | `2013-11-26 00:03:58.032` | Tracking           | ✅   | ✅      |
|                  | `etl_tstamp`          | `timestamp`              | Timestamp for when the event was validated and enriched     | No    | `2017-01-26 00:01:25.292` | Pipeline           | ✅   | ✅      |
|                  | `derived_tstamp`      | `timestamp`              | Timestamp making allowance for inaccurate device clock      | No    | `2013-11-26 00:02:04.123` | Default enrichment | ✅   | ✅      |
| `ttm`            | `true_tstamp`         | `timestamp`              | User-set "true timestamp" for the event                     | No    | `2013-11-26 00:02:04.123` | Tracking           | ✅   | ✅      |
|                  | `load_tstamp`         | `timestamp`              | Timestamp for when the data was loaded into the warehouse   | No    | `2013-11-26 00:02:04.123` | Pipeline           | ✅   | ✅      |
| `tz`             | `os_timezone`         | `string`, max length 255 | Client operating system timezone                            | No    | `Europe/London`           | Tracking           | ✅   | ✅      |

The trackers send timestamp properties as `int` in the payload, representing milliseconds since the Unix epoch, e.g. `1361553733313`. They're converted to `timestamp` type during enrichment. Similarly, the `tz` property is URL-encoded in the payload, e.g. `Europe%2FLondon`, and decoded during enrichment.

Read more about Snowplow timestamps [here](/docs/events/timestamps/).

### Device fields

The `dvce_screenheight` and `dvce_screenwidth` screen resolution fields originate from a single `res` payload property. It's a string with the format `"<width>x<height>"`, e.g. `1900x1024`. During enrichment, this string is split into the two separate integer fields.

| Payload property | Field name          | Type      | Description             | Reqd? | Example | Source   | Web | Mobile |
| ---------------- | ------------------- | --------- | ----------------------- | ----- | ------- | -------- | --- | ------ |
| `res`            | `dvce_screenheight` | `integer` | Screen height in pixels | No    | `1024`  | Tracking | ✅   | ✅      |
| `res`            | `dvce_screenwidth`  | `integer` | Screen width in pixels  | No    | `1900`  | Tracking | ✅   | ✅      |

For more information on this topic check out the [device data](/docs/events/ootb-data/device-and-browser/) page.

### IP address fields

These fields are populated by the [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/), based on the `user_ipaddress` field.

| Field name        | Type                     | Description                                                                                | Reqd? | Example              | Source                                                                                  | Web | Mobile |
| ----------------- | ------------------------ | ------------------------------------------------------------------------------------------ | ----- | -------------------- | --------------------------------------------------------------------------------------- | --- | ------ |
| `ip_isp`          | `string`, max length 100 | User's ISP                                                                                 | No    | `FDN Communications` | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `ip_organization` | `string`, max length 128 | Organization associated with the user's IP address - defaults to ISP name if none is found | No    | `Bouygues Telecom`   | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `ip_domain`       | `string`, max length 128 | Second level domain name associated with the user's IP address                             | No    | `nuvox.net`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `ip_netspeed`     | `string`, max length 100 | User's connection type                                                                     | No    | `Cable/DSL`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |

### Location fields

These fields are populated by the [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/), based on the IP address.

| Field name        | Type                     | Description                                            | Reqd? | Example              | Source                                                                                  | Web | Mobile |
| ----------------- | ------------------------ | ------------------------------------------------------ | ----- | -------------------- | --------------------------------------------------------------------------------------- | --- | ------ |
| `geo_country`     | `string`, max length 2   | ISO 3166-1 code for the country the user is located in | No    | `GB`, `US`           | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_region`      | `string`, max length 3   | ISO-3166-2 code for country region the user is in      | No    | `I9`, `TX`           | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_city`        | `string`, max length 75  | City the user is in                                    | No    | `New York`, `London` | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_zipcode`     | `string`, max length 15  | Postcode the user is in                                | No    | `94109`              | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_latitude`    | `float`                  | User location latitude                                 | No    | `37.443604`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_longitude`   | `float`                  | User location longitude                                | No    | `-122.4124`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_region_name` | `string`, max length 100 | User region name                                       | No    | `Florida`            | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_timezone`    | `string`, max length 64  | User timezone name                                     | No    | `Europe/London`      | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |

For more information on this topic check out the [geolocation data](/docs/events/ootb-data/geolocation/) page.

### Marketing fields

These fields are populated by the [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/).

| Field name     | Type                      | Description                                                 | Reqd? | Example                                 | Source                                                                                                               | Web | Mobile |
| -------------- | ------------------------- | ----------------------------------------------------------- | ----- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --- | ------ |
| `mkt_medium`   | `string`, max length 255  | Type of traffic source                                      | No    | 'cpc', 'affiliate', 'organic', 'social' | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_source`   | `string`, max length 255  | The company or website where the traffic came from          | No    | 'Google', 'Facebook'                    | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_term`     | `string`, max length 255  | Any keywords associated with the referrer                   | No    | 'new age tarot decks'                   | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_content`  | `string`, max length 500  | The content of the ad, or an ID so that it can be looked up | No    | 13894723                                | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_campaign` | `string`, max length 255  | The campaign ID                                             | No    | 'diageo-123'                            | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_clickid`  | `string`, max length 1000 | The click ID                                                | No    | 'ac3d8e459'                             | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_network`  | `string`, max length 64   | The ad network to which the click ID belongs                | No    | 'DoubleClick'                           | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |

### Cross-domain tracking fields

The pipeline populates these fields by default if the event's `url` has a [cross-navigation querystring](/docs/events/cross-navigation/).

| Payload property | Field name           | Type                     | Description                                                   | Reqd? | Example               | Source             | Web | Mobile |
| ---------------- | -------------------- | ------------------------ | ------------------------------------------------------------- | ----- | --------------------- | ------------------ | --- | ------ |
|                  | `refr_domain_userid` | `string`, max length 128 | The Snowplow `domain_userid` of the referring website         | No    | `bc2e92ec6c204a14`    | Default enrichment | ✅   | ✅      |
|                  | `refr_dvce_tstamp`   | timestamp                | The time of attaching the `domain_userid` to the inbound link | No    | `2013-11-26 00:02:05` | Default enrichment | ✅   | ✅      |

### Snowplow versions fields

These fields record the versions of the various Snowplow components involved in processing the event.

| Payload property | Field name    | Type                     | Description              | Reqd? | Example                             | Source             | Web | Mobile |
| ---------------- | ------------- | ------------------------ | ------------------------ | ----- | ----------------------------------- | ------------------ | --- | ------ |
| `tv`             | `v_tracker`   | `string`, max length 100 | Tracker version          | Yes   | `js-3.0.0`                          | Tracking           | ✅   | ✅      |
|                  | `v_collector` | `string`, max length 100 | Collector version        | Yes   | `ssc-2.1.0-kinesis`                 | Pipeline           | ✅   | ✅      |
|                  | `v_etl`       | `string`, max length 100 | ETL application versions | Yes   | `snowplow-micro-1.1.0-common-1.4.2` | Default enrichment | ✅   | ✅      |

## Web-specific fields

These fields apply mainly to events from web.

For more information about these fields check out the [device and browser](/docs/events/ootb-data/device-and-browser/), [page and screen view](/docs/events/ootb-data/page-and-screen-view-events/), and [links and referrers](/docs/events/ootb-data/links-and-referrers/) overview pages.

### Page fields

The `url`, `refr` and `page` payload properties are URL-encoded in the payload, e.g. `https%3A%2F%2Fwww.snowplow.io%2F` or `Snowplow%20Behavoral%20Data`, and decoded during enrichment.

The `url` property is further parsed during enrichment to populate the various `page_url*` fields. Similarly, the `refr` property is parsed to populate the various `refr_*` fields.

| Payload property | Field name         | Type                       | Description                                                            | Reqd? | Example                                                             | Source                                                                                                     | Web | Mobile |
| ---------------- | ------------------ | -------------------------- | ---------------------------------------------------------------------- | ----- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | --- | ------ |
| `url`            | `page_url`         | `string`, max length 10000 | The page URL                                                           | No    | `https://www.snowplow.io/product/index.html?id=GTM-DLRG#conclusion` | Tracking                                                                                                   | ✅   | ❌      |
|                  | `page_urlscheme`   | `string`, max length 16    | Scheme, also known as protocol                                         | No    | `https`                                                             | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlhost`     | `string`, max length 255   | Host, also known as domain                                             | No    | `www.snowplow.io`                                                   | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlport`     | `integer`                  | Port if specified, scheme dependent if not; 443 for HTTPS, 80 for HTTP | No    | `443`                                                               | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlpath`     | `string`, max length 3000  | Path to page                                                           | No    | `/product/index.html`                                               | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlquery`    | `string`, max length 6000  | Querystring                                                            | No    | `id=GTM-DLRG`                                                       | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlfragment` | `string`, max length 3000  | Fragment, also known as anchor                                         | No    | `conclusion`                                                        | Default enrichment                                                                                         | ✅   | ❌      |
| `page`           | `page_title`       | `string`, max length 2000  | Web page title                                                         | No    | `Snowplow Behavioral Data`                                          | Tracking                                                                                                   | ✅   | ❌      |
| `refr`           | `page_referrer`    | `string`, max length 10000 | URL of the referrer                                                    | No    | `https://www.google.com/search?q=psychic+oracle+cards`              | Tracking                                                                                                   | ✅   | ❌      |
|                  | `refr_urlscheme`   | `string`, max length 16    | Referrer scheme                                                        | No    | `https`                                                             | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlhost`     | `string`, max length 255   | Referrer host                                                          | No    | `www.google.com`                                                    | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlport`     | `integer`                  | Referrer port                                                          | No    | `443`                                                               | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlpath`     | `string`, max length 6000  | Referrer page path                                                     | No    | `/search`                                                           | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlquery`    | `string`, max length 6000  | Referrer URL querystring                                               | No    | `q=psychic+oracle+cards`                                            | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlfragment` | `string`, max length 3000  | Referrer URL fragment                                                  | No    |                                                                     | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_medium`      | `string`, max length 25    | Type of referrer                                                       | No    | `search`                                                            | [Referrer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) | ✅   | ❌      |
|                  | `refr_source`      | `string`, max length 50    | Name of referrer if recognized                                         | No    | `Google`                                                            | [Referrer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) | ✅   | ❌      |
|                  | `refr_term`        | `string`, max length 255   | Keywords if source is a search engine                                  | No    | `psychic oracle cards`                                              | [Referrer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) | ✅   | ❌      |

### Document fields

The `doc_width` and `doc_height` page size fields originate from a single `ds` payload property. It's a string with the format `"<width>x<height>"`, e.g. `3000x1024`. During enrichment, this string is split into the two separate integer fields.

| Payload property | Field name    | Type                     | Description                   | Reqd? | Example | Source   | Web | Mobile |
| ---------------- | ------------- | ------------------------ | ----------------------------- | ----- | ------- | -------- | --- | ------ |
| `cs`             | `doc_charset` | `string`, max length 128 | The page's character encoding | No    | `UTF-8` | Tracking | ✅   | ❌      |
| `ds`             | `doc_width`   | `integer`                | The page's width in pixels    | No    | `1024`  | Tracking | ✅   | ❌      |
| `ds`             | `doc_height`  | `integer`                | The page's height in pixels   | No    | `3000`  | Tracking | ✅   | ❌      |

### Browser fields

The `br_viewwidth` and `br_viewheight` page size fields originate from a single `vp` payload property. It's a string with the format `"<width>x<height>"`, e.g. `1000x1000`. During enrichment, this string is split into the two separate integer fields.

| Payload property | Field name      | Type                      | Description                              | Reqd? | Example                                                                                | Source               | Web | Mobile |
| ---------------- | --------------- | ------------------------- | ---------------------------------------- | ----- | -------------------------------------------------------------------------------------- | -------------------- | --- | ------ |
| `lang`           | `br_lang`       | `string`, max length 255  | Language the browser is set to           | No    | `en-GB`                                                                                | Tracking             | ✅   | ❌      |
| `cookie`         | `br_cookies`    | `integer boolean`         | Whether the browser is accepting cookies | No    | `1`                                                                                    | Tracking             | ✅   | ❌      |
| `cd`             | `br_colordepth` | `string`, max length 12   | Bit depth of the browser color palette   | No    | `24`                                                                                   | Tracking             | ✅   | ❌      |
| `vp`             | `br_viewwidth`  | `integer`                 | Viewport width                           | No    | `1000`                                                                                 | Tracking             | ✅   | ❌      |
| `vp`             | `br_viewheight` | `integer`                 | Viewport height                          | No    | `1000`                                                                                 | Tracking             | ✅   | ❌      |
| `ua`             | `useragent`     | `string`, max length 1000 | Raw useragent                            | No    | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0` | Tracking or pipeline | ✅   | ✅      |

## Baked-in event fields

[Baked-in event](/docs/fundamentals/events/#baked-in-events) data is stored in specific default fields, rather than in self-describing event or entity columns.

### Page views

There are no fields that are specific to page view events. All the relevant fields are included in the standard fields available for any web-based event e.g. `page_url`, or `page_title`.

### Page pings

Page ping events include four additional fields. They indicate how a user has scrolled over a web page since the last page ping:

| Payload property | Field name       | Type      | Description                                        | Reqd? | Example |
| ---------------- | ---------------- | --------- | -------------------------------------------------- | ----- | ------- |
| `pp_mix`         | `pp_xoffset_min` | `integer` | Minimum page x offset seen in the last ping period | No    | `10`    |
| `pp_max`         | `pp_xoffset_max` | `integer` | Maximum page x offset seen in the last ping period | No    | `100`   |
| `pp_miy`         | `pp_yoffset_min` | `integer` | Minimum page y offset seen in the last ping period | No    | `5`     |
| `pp_may`         | `pp_yoffset_max` | `integer` | Maximum page y offset seen in the last ping period | No    | `200`   |

### Structured events

Structured events use these fields to capture event data:

| Payload property | Field name    | Type                      | Description                                                                                     | Reqd?  | Example                       |
| ---------------- | ------------- | ------------------------- | ----------------------------------------------------------------------------------------------- | ------ | ----------------------------- |
| `se_ca`          | `se_category` | `string`, max length 1000 | Category of event                                                                               | Yes \* | `ecomm`, `video`              |
| `se_ac`          | `se_action`   | `string`, max length 1000 | Action performed, or event name                                                                 | Yes \* | `add-to-basket`, `play-video` |
| `se_la`          | `se_label`    | `string`, max length 4096 | The object of the action e.g. the ID of the video played. or SKU of the product added to basket | No     | `pbz00123`                    |
| `se_pr`          | `se_property` | `string`, max length 1000 | A property associated with the object of the action                                             | No     | `HD`, `large`                 |
| `se_va`          | `se_value`    | `decimal`                 | A value associated with the event or action e.g. the value of goods added-to-basket             | No     | `9.99`                        |

\* These fields are only required for structured events.

## Self-describing event fields

For [self-describing events](/docs/fundamentals/events/#self-describing-events), the pipeline extracts data from the event schema and populates the following fields:

| Field name      | Type                      | Description                           | Reqd? | Example      | Source             | Web | Mobile |
| --------------- | ------------------------- | ------------------------------------- | ----- | ------------ | ------------------ | --- | ------ |
| `event_vendor`  | `string`, max length 1000 | Business that defined the event       | Yes   | `com.acme`   | Default enrichment | ✅   | ✅      |
| `event_name`    | `string`, max length 1000 | Event name                            | Yes   | `link_click` | Default enrichment | ✅   | ✅      |
| `event_format`  | `string`, max length 128  | Format for event; always `jsonschema` | Yes   | `jsonschema` | Default enrichment | ✅   | ✅      |
| `event_version` | `string`, max length 128  | Version of event schema               | Yes   | `1-0-2`      | Default enrichment | ✅   | ✅      |

> **Note:** Data in self-describing events or entities is added as additional columns, rather than in these standard atomic fields.
>
> Check out the [warehouse table structure](/docs/fundamentals/warehouse-tables/) page to learn more.

## Deprecated fields

We have deprecated the following fields. They're no longer populated by Snowplow trackers or enrichments, but they're still part of the Snowplow tracker protocol:

| Field name                 | Type                     | Description                                                                | Reqd? | Example      | Source     | Web | Mobile |
| -------------------------- | ------------------------ | -------------------------------------------------------------------------- | ----- | ------------ | ---------- | --- | ------ |
| `txn_id`                   | `integer`                | Transaction ID set client-side, used to de-dupe records                    | No    | `421828`     | Deprecated | ✅   | ✅      |
| `dvce_type`                | `string`, max length 50  | Type of device                                                             | No    | `Computer`   | Deprecated | ✅   | ✅      |
| `dvce_ismobile`            | `integer boolean`        | Is the device mobile?                                                      | No    | `1`          | Deprecated | ✅   | ✅      |
| `os_name`                  | `string`, max length 50  | Name of operating system                                                   | No    | `Android`    | Deprecated | ✅   | ✅      |
| `os_family`                | `string`, max length 50  | Operating system family                                                    | No    | `Linux`      | Deprecated | ✅   | ✅      |
| `os_manufacturer`          | `string`, max length 50  | Company responsible for OS                                                 | No    | `Apple`      | Deprecated | ✅   | ✅      |
| `user_fingerprint`         | `string`, max length 128 | A user fingerprint generated by looking at the individual browser features | No    | `2161814971` | Deprecated | ❌   | ❌      |
| `br_name`                  | `string`, max length 50  | Browser name                                                               | No    | `Firefox 12` | Deprecated | ❌   | ❌      |
| `br_version`               | `string`, max length 50  | Browser version                                                            | No    | `12.0`       | Deprecated | ❌   | ❌      |
| `br_family`                | `string`, max length 50  | Browser family                                                             | No    | `Firefox`    | Deprecated | ❌   | ❌      |
| `br_type`                  | `string`, max length 50  | Browser type                                                               | No    | `Browser`    | Deprecated | ❌   | ❌      |
| `br_renderengine`          | `string`, max length 50  | Browser rendering engine                                                   | No    | `GECKO`      | Deprecated | ❌   | ❌      |
| `br_features_pdf`          | `integer boolean`        | Whether the browser recognizes PDFs                                        | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_flash`        | `integer boolean`        | Whether Flash is installed                                                 | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_java`         | `integer boolean`        | Whether Java is installed                                                  | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_director`     | `integer boolean`        | Whether Adobe Shockwave is installed                                       | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_quicktime`    | `integer boolean`        | Whether QuickTime is installed                                             | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_realplayer`   | `integer boolean`        | Whether RealPlayer is installed                                            | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_windowsmedia` | `integer boolean`        | Whether mplayer2 is installed                                              | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_gears`        | `integer boolean`        | Whether Google Gears is installed                                          | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_silverlight`  | `integer boolean`        | Whether Microsoft Silverlight is installed                                 | No    | `1`          | Deprecated | ✅   | ❌      |

### Legacy ecommerce fields

There are a large number of fields in the tracker protocol specifically for [legacy ecommerce](/docs/fundamentals/events/#baked-in-events) `transaction` and `transaction_item` events.

Fields that start `tr_` relate to the transaction as a whole. Fields that start `ti_` refer to the specific item included in the transaction, e.g. a product in the basket.

Single transactions typically span multiple lines of data: there will be a single line where `event` = `transaction`, where the `tr_` fields are set, and multiple lines (one for each product included) where `event` = `transaction_item` and the `ti_` fields are set.

| Field name            | Type                     | Description                                                | Reqd? | Example         |
| --------------------- | ------------------------ | ---------------------------------------------------------- | ----- | --------------- |
| `tr_orderid`          | `string`, max length 255 | Order ID                                                   | Yes   | `#134`          |
| `tr_affiliation`      | `string`, max length 255 | Transaction affiliation (e.g. store where sale took place) | No    | `web`           |
| `tr_total`            | `decimal`                | Total transaction value                                    | Yes   | `12.99`         |
| `tr_tax`              | `decimal`                | Total tax included in transaction value                    | No    | `3.00`          |
| `tr_shipping`         | `decimal`                | Delivery cost charged                                      | No    | `0.00`          |
| `tr_total_base` \*    | `decimal`                | Total in base currency                                     | No    | `12.99`         |
| `tr_tax_base` \*      | `decimal`                | Total tax in base currency                                 | No    | `3.00`          |
| `tr_shipping_base` \* | `decimal`                | Delivery cost in base currency                             | No    | `0.00`          |
| `tr_city`             | `string`, max length 255 | Delivery address, city                                     | No    | `London`        |
| `tr_state`            | `string`, max length 255 | Delivery address, state                                    | No    | `Washington`    |
| `tr_country`          | `string`, max length 255 | Delivery address, country                                  | No    | `France`        |
| `tr_currency`         | `string`, max length 3   | Currency                                                   | No    | `USD`           |
| `ti_orderid`          | `string`, max length 255 | Order ID                                                   | Yes   | `#134`          |
| `ti_sku`              | `string`, max length 255 | Product SKU                                                | Yes   | `pbz00123`      |
| `ti_name`             | `string`, max length 255 | Product name                                               | No    | `Cone pendulum` |
| `ti_category`         | `string`, max length 255 | Product category                                           | No    | `New Age`       |
| `ti_price`            | `decimal`                | Product unit price                                         | Yes   | `9.99`          |
| `ti_price_base` \*    | `decimal`                | Price in base currency                                     | No    | `9.99`          |
| `ti_quantity`         | `integer`                | Number of product in transaction                           | Yes   | `2`             |
| `ti_currency`         | `string`, max length 3   | Currency                                                   | No    | `EUR`           |
| `base_currency` \*    | `string`, max length 3   | Reporting currency                                         | No    | `GBP`           |

\* Set exclusively by the [Currency conversion enrichment](/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/).

---

# Introduction to data destinations
> Different options for storing and forwarding Snowplow data.
> Source: https://docs.snowplow.io/docs/fundamentals/destinations/

![Diagram showing Snowplow data flowing from a data warehouse into three destination categories: Event Forwarding (customer journey nudges, packaged analytics), Reverse ETL (targeting in Google/Facebook, CRM enablement, packaged analytics), and BI and ML Tool (in-session recommendations, customer-facing metrics, churn propensity scoring, fraud prevention, custom marketing attribution, content and author analytics, lead scoring, anomaly detection)](/assets/images/usecasearch-5b84782067868dc4be86b6012cf315ea.png)

Snowplow gives you multiple ways to get your data where it needs to go. You can load data into warehouses and lakes for deeper analysis, forward individual events to marketing platforms in real-time, or send aggregated segments to activate your insights.

The right approach depends on your use case, but most organizations benefit from combining approaches: storing data centrally in a warehouse while selectively forwarding specific events or segments to operational systems.

## Data warehouses and lakes

Snowplow loads your data into warehouses and lakes where you can combine it with other datasets for analysis. You can choose from major data platforms including Redshift, BigQuery, Snowflake, Databricks, and Synapse Analytics, as well as cloud storage like S3, GCS, and ADLS through [our various loaders](/docs/destinations/warehouses-lakes/).

## Event forwarding

Send individual events to downstream platforms in real-time through [event forwarding](/docs/destinations/forwarding-events/). This approach works well for:

- Marketing and personalization platforms that can act on fresh event data
- Product analytics tools that require granular event-level data
- Streaming platforms like Kafka, Kinesis, and Pub/Sub for internal consumption and real-time processing

## Reverse ETL

Sync aggregated data and customer segments from your warehouse to operational systems through [reverse ETL](/docs/destinations/reverse-etl/). This approach lets you activate insights from your data by sending calculated metrics, audience segments, or enriched profiles to marketing and sales tools.

Snowplow recommends [Fivetran Activations](https://www.fivetran.com/data-movement/activations) for reverse ETL.

---

# Introduction to Snowplow entities
> Entities are a good way to deal with common fields across various events
> Source: https://docs.snowplow.io/docs/fundamentals/entities/

When you track an action or behavior, the information about the objects, users, and context in which the action occurred is just as important as the action itself. In Snowplow tracking, we use entities to capture this contextual data. They're reusable building blocks that make your tracking easier to implement, and your data easier to analyze.

> **Info:** What we now call "entity" or "entities" was previously called "context". You'll still find `context` or `contexts` used in many of the existing APIs, database column names, and documentation, especially to refer to a set of multiple entities.

For example, when tracking a "search" event, you might want to capture information about:

- The user who performed the search
- The web page where the search was made
- The products that were returned in the search results
- Any A/B test variations that were active on the page
- And so on

The event data might include fields such as the search term, number of results, and time taken to perform the search. These fields are specific to the "search" event.

The `user`, `webPage`, `A/B test variation`, and `products` data isn't specific to that event, and can be defined and tracked as entities:

![Diagram showing an example Snowplow event with atomic event data at the top, self-describing event data in the middle, and entities at the bottom including user, webPage, abTestID, and three product entities](/assets/images/example-event-entities-9ddd43c1ce9bd436594766b8067fdb6f.png)

The user who triggered the `search` event might go on to generate a `link_click` event. You might want to capture data on:

- The user who clicked the link
- The web page where the link was clicked

The same `user` and `webPage` entities can be reused for this event.

Every Snowplow event can have entities attached to it. They can be of the same or different types.

## How to track entities

Snowplow provides a number of entities out-of-the-box. You can configure your trackers to attach certain entities automatically to all events. For example, the web trackers add a [`webPage` entity](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) to all events by default.

Trackers add other out-of-the-box entities based on event type. For example, if you track a Snowplow media `play` event, the tracker will automatically add `media_player` and media `session` entities to it.

Check out the [out-of-the-box data](/docs/events/ootb-data/) section to find out more about the entities that come with Snowplow trackers.

Some [enrichments](/docs/pipeline/enrichments/available-enrichments/) add entities to events in stream. For example, the [cross-navigation enrichment](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/) adds a `cross_navigation` entity to events based on the cross-navigation querystring.

### Custom entities

Check out our [tracking plan guide](/docs/fundamentals/tracking-design-best-practice/) for best practice on designing entities.

Snowplow provides [tooling](/docs/event-studio/) to help you define your own custom entities. Read more about tracking custom data [here](/docs/events/custom-events/).

## Entities are defined by schemas

Like most Snowplow [events](/docs/fundamentals/events/#self-describing-events), entities are based on [self-describing JSON schemas](/docs/fundamentals/schemas/). As such, they can include arbitrarily complex data, and are versioned. You can evolve your entity definitions over time.

Each entity consists of two parts:

- `schema`: a reference to a [schema](/docs/fundamentals/schemas/) that describes the name, version and structure of the entity
- `data`: the entity data as a set of key-value properties in JSON format

Find out in the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page about how entities are structured in the data warehouse.

---

# Introduction to Snowplow events
> An event is a central concept in Snowplow that represents something that occurred at a particular point in time
> Source: https://docs.snowplow.io/docs/fundamentals/events/

Events are the primary way to capture data with Snowplow. Every time something happens that you want to track, you can send an event to your Snowplow pipeline. Each event is a JSON object that describes what happened. Events flow through your pipeline where they're validated, enriched, and loaded into your data warehouse or lake for analysis.

## Snowplow event types

All Snowplow events have the same underlying structure and [standard fields](/docs/fundamentals/canonical-event/). However, there are two types of events in Snowplow:

- Schema-less **baked-in** events
  - The baked-in event types are page views, page pings, and structured events

- **Self-describing** events that are based on a [JSON schema](/docs/fundamentals/schemas/)

  - All other events are self-describing events
  - This includes out-of-the-box events that come with Snowplow, and custom events that you define yourself

![Diagram showing the different types of Snowplow event](/assets/images/Snowplow-event-types-99d78b7b2932b01b0165b2990fb3085e.png)

Whether an event is baked-in or self-describing affects how you'll model the data.

All Snowplow events are by default tracked with schema-defined [entities](/docs/fundamentals/entities/) attached to them.

### Baked-in events

The following events are "baked in":

- [Page views](/docs/events/ootb-data/page-and-screen-view-events/)
- [Page pings](/docs/events/ootb-data/page-activity-tracking/)
- [Structured events](/docs/events/custom-events/#structured-events)

In the data warehouse, any event-specific information will be in standard columns in the Snowplow `events` table. You can find those listed [here](/docs/fundamentals/canonical-event/#baked-in-event-fields).

Find out more about how to track and model page view and page ping events [here](/docs/events/ootb-data/page-activity-tracking/).

> **Tip:** We recommend using self-describing events instead of structured events whenever possible. Structured event tracking is a legacy format used to track events that were not natively supported by Snowplow.

> **Info:** The legacy `transaction` and `transaction_item` events are also "baked-in". Several Snowplow trackers provide [ecommerce tracking](/docs/events/ootb-data/ecommerce-events/) APIs; we recommend using these instead, as they support current Snowplow best practise, and are compatible with the [Snowplow ecommerce dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/).

### Self-describing events

Self-describing events can include arbitrarily complex data, as defined by the event's [data structure](/docs/fundamentals/schemas/) or schema. We call them "self-describing" because these events include a reference to their underlying [JSON Schema](/docs/api-reference/json-schema-reference/) data structure.

Each self-describing event consists of two parts:

- `schema`: a reference to a [schema](/docs/fundamentals/schemas/) that describes the name, version and structure of the event
- `data`: the event data as a set of key-value properties in JSON format

Because the event references a specific version of its schema, it's always clear to downstream users and applications what each field in the event means, even if your definition of the event changes over time.

Snowplow provides a large number of self-describing events out-of-the-box, for example:

- [Link clicks](/docs/events/ootb-data/links-and-referrers/#link-clicks)
- [Form submissions](/docs/sources/web-trackers/tracking-events/form-tracking/)
- [Ecommerce transactions](/docs/events/ootb-data/ecommerce-events/)

Check out the full range of included events [here](/docs/events/ootb-data/).

You can also create [custom self-describing events](/docs/events/custom-events/) to match your business requirements. Snowplow provides [tooling](/docs/event-studio/) to help you define and track custom events.

Find out in the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page about how self-describing events are structured in the data warehouse.

> **Info:** We originally called self-describing events "unstructured events", to distinguish them from structured events. This was misleading, because these events are actually more structured than structured events. The old term is deprecated, but you might still see it in some docs, APIs and database column names, such as `unstruct_event` or `ue`.

## How to track events

Track events in your applications using one of the [Snowplow tracking SDKs](/docs/sources/). Each SDK provides built-in APIs to track [different types of events](/docs/events/ootb-data/).

All the trackers also support custom tracking, so you can define the events that are relevant to your business.

You can also use [webhooks](/docs/sources/webhooks/) to track automated actions. The Snowplow [Collector endpoint](/docs/pipeline/collector/) accepts all [valid](/docs/fundamentals/canonical-event/) Snowplow event payloads, regardless of their source.

---

# Introduction to failed events
> Failed events represent data that did not pass validation or otherwise failed to be processed
> Source: https://docs.snowplow.io/docs/fundamentals/failed-events/

_Failed events_ is an umbrella term for events that the pipeline had some problem processing.

These problems can arise at various stages:

- Collection (e.g. invalid payload format)
- Validation (e.g. event does not match the schema)
- Enrichment (e.g. external API unavailable)
- Loading (very unlikely with modern versions of Snowplow)

> **Note:** Failed events are **not** written to your `atomic` events table, which only contains high quality data. See [below](#dealing-with-failed-events) for how to deal with them.

## Common failures

The two most common types of failed events are:

- **Validation failures**. These errors occur when an [event](/docs/fundamentals/events/) or an [entity](/docs/fundamentals/entities/) does not match its [schema](/docs/fundamentals/schemas/). The reason usually is incorrect tracking code. For example, if you have a schema with a property defined as an `enum` accepting values `a`, `b`, and `c`, but the incoming event which references that schema includes the property with a value of `d`, that will be a `ValidationError`. The error's description will include that information -- e.g. `$property does not match enum list`. Validation can also fail if the schema is not available, such as when you forgot to add it to the production schema registry before putting the tracking code live. The pipeline will first search for a schema in the associated [Iglu](/docs/fundamentals/schemas/#iglu-schema-repository) repository, and if it cannot find it, it will search in [Iglu Central](https://iglucentral.com), the public repository of self-describing schemas available to everyone. If the schema is not found in either location, the event will be labeled as a `ResolutionError`.

- **Enrichment failures**. These can be due to an [API enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/) reaching out to an external API that’s down. Another cause is a failure in the custom code in a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/).

> **Tip:** In many cases, you will be able to fix the underlying problem directly, e.g. by altering your tracking code, by providing the correct schema, or by changing your enrichment configuration.

## Other failures

Other failures generally fall into 3 categories:

- **Bots or malicious activity**. Bots, vulnerability scans, and so on, can send completely invalid events to the Collector. The format might be wrong, or the payload size might be extraordinarily large.

- **Pipeline misconfiguration**. For example, a loader could be reading from the wrong stream (with events in the wrong format). This is quite rare, unless you're self-hosting Snowplow, as all relevant pipeline configuration is automatic.

- **Temporary infrastructure issue**. This is again rare. One example would be Iglu Server (schema registry) not being available.

> **Tip:** All of these are internal failures you typically can’t address upstream.

## Dealing with failed events

Snowplow provides a dashboard and alerts for failed events. See [Monitoring failed events](/docs/monitoring/).

***

For the common failures (validation and enrichment), you can configure continuous loading of any offending events into _a separate table_ in your warehouse or lake. This way, you can easily inspect them and decide how they might be patched up (e.g. with SQL) and merged with the rest of your data.

> **Note:** This feature is not retroactive, i.e. only failed events that occur _after it’s enabled_ will be loaded into your desired destination.

The events will include a special column with the details of the failure, and any invalid columns will be set to `null`. Otherwise, the format is [the same as for your atomic events](/docs/fundamentals/canonical-event/).

See [Exploring failed events](/docs/monitoring/exploring-failed-events/) for more details and setup instructions.

***

Finally, on AWS and GCP all failed events are backed up in object storage (S3 and GCS respectively). Sometimes, but not in all cases (e.g. not if the original events exceeded size limits), it’s possible to recover them by replaying them through the pipeline. This is a complicated process mainly reserved for internal failures and outages. Refer to [Recovering failed events](/docs/monitoring/recovering-failed-events/).

***

You can find a full list of failed event types [in the API reference section](/docs/api-reference/failed-events/).

---

# Snowplow fundamentals
> An overview of what a Snowplow pipeline does and what steps it includes
> Source: https://docs.snowplow.io/docs/fundamentals/

![Snowplow pipeline architecture diagram showing data flowing from web, mobile, server-side, pixel, and webhook sources through Collect, Validate, and Enrich stages. Validated events route to a Real-Time Good Stream and on to a data warehouse and BI tools; invalid events route to a Real-Time Bad Stream and Bad Logs.](/assets/images/architecture-16e4cd694721410bb8a1ff943a4c5255.png)

The diagram above illustrates a typical Snowplow pipeline with data flowing left to right.

- [**Trackers**](/docs/sources/) generate [event](/docs/fundamentals/events/) data and send this to your Collector. We have trackers covering web, mobile, desktop, server and IoT. Additionally, [**webhooks**](/docs/sources/webhooks/) allow third-party software to send their own internal event streams to your Collector for further processing.
- Events hit the **Collector** application and it saves the raw events to storage (S3 on AWS and Google Cloud Storage on GCP) and then sends them down the pipeline to Enrich.
- The **Enrich** application cleanses the data and validates each event against its [schema](/docs/fundamentals/schemas/) to ensure it meets the criteria you have designed and set. When an event fails to validate, it will feed into a bad data stream which contains all of your [**failed events**](/docs/fundamentals/failed-events/). This way, the Snowplow pipeline is non-lossy as failed events [can be reprocessed](/docs/monitoring/recovering-failed-events/).
- Once validated, each event is enriched by the [**Enrichments**](/docs/pipeline/enrichments/available-enrichments/) you have configured for your pipeline.
- The enriched events are sent to real-time streams (Kinesis on AWS, PubSub on GCP). At this point, data can be [**forwarded**](/docs/destinations/forwarding-events/) to other applications to power real-time use cases.
- [**Loaders**](/docs/destinations/warehouses-lakes/) then load your data off the real-time streams into the various destinations you have set up for your pipeline. Typically these include a data warehouse (Redshift, Snowflake, BigQuery, Databricks) or a data lake (S3, GCS, ADLS / OneLake). Note that you can also connect a data warehouse to a data lake (e.g. Synapse Analytics + ADLS) instead of loading to a warehouse directly.
- Your event-level data is now in your chosen destinations. From here we recommend this raw data is [**modelled**](/docs/modeling-your-data/) into smaller, cleaner tables that are easier to perform analysis on. At this stage it can also be joined with other data sets. We have out-of-the-box data models for web and mobile to help you get started.

## Why do we use this architecture?

Snowplow's distinctive architecture has been informed by a set of key design principles:

1. **Extreme scalability** - Snowplow should be able to scale to tracking billions of customer events without affecting the performance of your client (e.g. website) or making it difficult to subsequently analyze all of those events
2. **Permanent event history** - Snowplow events should be stored in a simple, non-relational, immutable data store. Your raw event data is never compacted, overwritten or otherwise corrupted by Snowplow
3. **Direct access to individual events** - you should have direct access to your raw Snowplow event data at the atomic level. Unlike with other solutions, the data is not intermediated by a third-party vendor, or a slow API, or an interface offering aggregates only
4. **Separation of concerns** - event tracking and event analysis should be two separate systems, only loosely-coupled. As a result, Snowplow can be used to feed whatever analytics process you want
5. **Support any analysis** - Snowplow should make it easy for business analysts, data scientists and engineers to answer any business question they want, using as wide a range of analytical tools as possible, including Machine Learning

---

# Introduction to structuring your data with schemas
> Schemas are a powerful feature that ensures your data is clean and descriptive
> Source: https://docs.snowplow.io/docs/fundamentals/schemas/

**Schemas** define the structure of the data that you collect. Each schema defines what fields are recorded with each [event](/docs/fundamentals/events/), and provides validation criteria for each field. Schemas are also used to describe the structure of [entities that are attached to events](/docs/fundamentals/entities/).

With schemas, you can:

- Build meaning into your data by clearly defining what each field represents.
- Define your own data structures to capture data in a way that works for your business. For example, a two-sided marketplace will be tracking very different events from a gaming application.
- Ensure your data presents a validated record of what has happened.

Schemas describe how you structure your data. When data is [processed through your Snowplow pipeline](/docs/fundamentals/), each event is validated against its schema and only valid events are allowed to pass through. [Failed events](/docs/fundamentals/failed-events/) are sent to a separate location.

By describing how the data should be structured as part of your schema definition, you ensure clean and consistent data landing in your data warehouse or other destinations.

As you evolve your website, mobile app, or server-side application, you can update your schemas to reflect the changes. Snowplow automatically evolves your table definition to accommodate both old and new data safely.

## Data structures are schemas plus metadata

We often use the terms "schema" and "data structure" interchangeably, but they're not exactly the same thing. A **schema** is the JSON Schema definition that describes the structure of your data.

A **data structure** is a higher-level data management concept that includes the schema along with additional metadata, including:

- Whether the schema is to be used for an event or an entity
- Whether it's available to your pipeline, or is still in draft
- Optional change notes associated with schema versions

[Data structures](/docs/event-studio/data-structures/) wrap schemas with this additional metadata to help you manage your data definitions. They allow you to group related schemas within [tracking plans](/docs/event-studio/tracking-plans/) (previously known as data products), or keep track of [which schemas are used where](/docs/event-studio/tracking-plans/event-specifications/).

> **Tip:** Check out the documentation for [managing](/docs/event-studio/data-structures/) and [versioning](/docs/event-studio/data-structures/versioning/) data structures.

## Self-describing JSON schema anatomy

> **Info:** Snowplow CDI customers can create custom schemas using the [data structures builder](/docs/event-studio/data-structures/), without worrying about how it works under the hood.

Snowplow schemas are based on the [JSON Schema](https://json-schema.org/) standard ([draft 4](https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00)). For a comprehensive guide to all Snowplow supported validation options, see the [Snowplow JSON Schema reference](/docs/api-reference/json-schema-reference/).

Snowplow specifically uses **self-describing JSON schemas**, meaning that they contain metadata about themselves within the schema definition.

Let's take a look at an example schema to talk about its constituent parts:

```json
{

  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

  "description": "Schema for an example event",

  "self": {

    "vendor": "com.snowplowanalytics",

    "name": "example_event",

    "format": "jsonschema",

    "version": "1-0-0"

  },

  "type": "object",

  "properties": {

    "name": {

      "type": "string",

      "maxLength": 128

    },

    "job_role": {

      "description": "",

      "type": [

        "string",

        "null"

      ],

      "maxLength": 128

    },

    "promo_code": {

      "description": "",

      "type": [

        "string",

        "null"

      ],

      "minLength": 8,

      "maxLength": 20

    }

  },

  "additionalProperties": false,

  "required": [

    "name"

  ]

}
```

`$schema`: this argument instructs the Snowplow pipeline on how to handle this schema and in most circumstances should be left as shown in the example.

`description`: this argument is where you should put detailed information on the purpose of this schema. This will be particularly helpful for others who are trying to understand the meaning of particular data or when they want to know if a schema already exists for something they want to track.

`self`: this section of arguments contains metadata which makes the schema "self-describing".

- `vendor`: this usually refers to the company who has authored the schema. Most times this will be your company’s name. This could also be for organizing schemas from different groups in your organization if you have multiple teams working on different events and entities (e.g. `com.acme.android`, `com.acme.marketing`). Snowplow uses the reversed company internet domain for vendor names (e.g. `com.snowplowanalytics`).
- `name`: this is the name you want to give your schema. Much like the description above, this is a good chance to help others like data analysts who might be consuming this data know exactly what your schema is meant to capture.
- `format`: this field simply states the format of the schema which will always be `jsonschema`.
- `version`: Snowplow allows you to [increment versions of a schema](/docs/event-studio/data-structures/versioning/) as your tracking needs to evolve and this argument stores the current version.

After the self section, the remainder of the schema is where you will begin describing the event or entity fields that you will be collecting.

`type`: type should always be set as `object`.

`properties`: here is where you will describe the fields you intend on collecting. Each field is given a name and a number of arguments, these are important as they feed directly into the validation process.

- `description`: similar to the description field for the schema, this argument is where you should put detailed information on what this field represents to avoid any misunderstanding or misinterpretation during analysis.
- `type`: this denotes the type of data that is collected through this field. The most common types of data collected are `string`, `number`, `integer`, `object`, `array`, `boolean` and `null`. A single field can allow multiple types as shown in the field `job role` in the example schema which allows both `string` and `null`
- Validation arguments can then be passed into the field such as `minLength`, `maxLength` and `enum` for strings and `minimum` and `maximum` for integers.

This example doesn't show the optional `$supersedes` and `$supersededBy` fields. See [marking schemas as superseded](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded).

## Iglu schema repository

> **Info:** Snowplow CDI customers don't need to create or manage their own Iglu repositories. A private Iglu repository is included in your pipeline.

**Iglu** is a set of tools for hosting and managing JSON schemas. We use it to store all the schemas associated with events and entities. It's called a schema repository or schema registry.

You'll notice that schema URIs use the `iglu:` protocol, for example: `iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0`. This is because all Snowplow schemas are stored in an Iglu repository.

There is a [central Iglu repository](https://iglucentral.com/) that holds public schemas for use with Snowplow, including ones for some of the [out-of-the-box self-describing events](/docs/fundamentals/events/#self-describing-events) and [out-of-the-box entities](/docs/fundamentals/entities/#how-to-track-entities).

For Snowplow Self-Hosted users, you'll need to run your own Iglu schema repository to host schemas for your custom [events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/#custom-entities). Use [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) (recommended), or a [static repository](/docs/api-reference/iglu/iglu-repositories/static-repo/).

## Self-describing JSON data in the event payload

Snowplow trackers can send self-describing JSON event and entity data as is, or as Base64-encoded strings. Base-64 strings are URL-safe and ensure that no data is lost or corrupted. The downside is that the data will be bigger and less readable.

Here's an example showing a self-describing JSON object for a product view event:

```json
{

  "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",

  "data": {

    "product_id": "ASO01043",

    "price": 49.95

  }

}
```

Before adding this data to the event payload, the tracker will wrap this event self-describing JSON in an outer self-describing JSON. The wrapper specifies that this is a self-describing event, using the historical `unstruct_event` naming:

```json
{

  // Tells Snowplow this is an self-describing event

  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",

  "data": {



    // Tells Snowplow this is a viewed_product event

    "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",

    "data": {



      // The event data itself

      "product_id": "ASO01043",

      "price": 49.95

    }

  }

}
```

> **Info:** The event payload is itself a self-describing JSON object. See how this works in practise in these [example tracker HTTP requests](/docs/events/http-requests/).

The parameter name used for this wrapped structure depends on whether you've chosen to use Base-64 encoding or not:

| Payload property | Type                | Example values                                                 |
| ---------------- | ------------------- | -------------------------------------------------------------- |
| `ue_px`          | Base64-encoded JSON | `eyAicHJvZHVjdF9pZCI6ICJBU08wMTA0MyIsICJwcmljZSI6IDQ5Ljk1IH0=` |
| `ue_pr`          | JSON                | `{ "product_id": "ASO01043", "price": 49.95 }`                 |

Trackers process entities in a similar way. The differences are:

- They're wrapped in a `contexts` self-describing JSON rather than `unstruct_event`
- Payload parameter options are `co` for JSON or `cx` for Base64-encoded JSON

See the [schema translation reference](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) page to learn how schema data lands in your warehouse.

---

# Versioning, patching, and marking schemas as superseded
> Fix mistakes in existing schema versions by amending them, with guidelines on when amendments are appropriate and how to handle production schemas.
> Source: https://docs.snowplow.io/docs/fundamentals/schemas/versioning/

Every schema has a version, starting from `1-0-0`. As your schema evolves, you will need to create new versions of it. All previous versions of a schema remain available to ensure backwards-compatibility.

Versioning has an important role in telling Snowplow Loaders how to handle the changes when loading into your data warehouse. For example, there may be a need to create new columns, update columns, or even create new tables.

It's important to understand when your change is breaking, and [version correctly](/docs/event-studio/data-structures/versioning/).

There are two kinds of schema changes:

- **Non-breaking** — a non-breaking change is backward compatible with historical data and increments the `patch` version (for example, `1-0-0` to `1-0-1`) or the `minor` version (for example, `1-0-0` to `1-1-0`).
- **Breaking** — a breaking change is not backward compatible with historical data and increments the `major` version (for example, `1-0-0` to `2-0-0`).

Different data warehouses handle schema evolution slightly differently. Use the table below as a guide for incrementing the schema version appropriately.

|                                              | Redshift     | Snowflake, BigQuery, Databricks |
| -------------------------------------------- | ------------ | ------------------------------- |
| **Add / remove / rename an optional field**  | Non-breaking | Non-breaking                    |
| **Add / remove / rename a required field**   | Breaking     | Breaking                        |
| **Change a field from optional to required** | Breaking     | Breaking                        |
| **Change a field from required to optional** | Breaking     | Non-breaking                    |
| **Change the type of an existing field**     | Breaking     | Breaking                        |
| **Change the size of an existing field**     | Non-breaking | Non-breaking                    |

> **Note:** In Redshift and Databricks, changing _size_ may also mean _type_ change. For example, changing the `maximum` integer from `30000` to `100000`. See our documentation on [how schemas translate to database types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

Changing a schema without incrementing the version has several risks:

- Events that were previously valid could become invalid against the new changes
- Your warehouse Loader, which updates the table [according to the schema](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/#versioning), could get stuck if it’s not possible to cast the data in the existing table column to the new definition (e.g. if you change a field type from a string to a number)
- Similarly, data models or other applications consuming the data downstream might not be able to deal with the changes

Aim to treat each schema version as immutable. However, if you can't avoid making changes to an existing schema version, you have two options: patching the schema, or marking it as superseded.

## Schema version format

Snowplow versions schemas using a scheme called **SchemaVer**. A SchemaVer version has the form `major-minor-patch`, starting at `1-0-0`:

- `major` — increment for a breaking change that prevents the new schema from validating historical data (for example, changing a field's type, or adding a new required field).
- `minor` — increment for a change that _may_ prevent the new schema from validating some historical data.
- `patch` — increment for a change that's compatible with all historical data (for example, adding a new optional field). Note that this is unrelated to overwriting a schema version, which is also called patching.

Unlike [SemVer](https://semver.org/), SchemaVer uses hyphens rather than periods, and starts at `1-0-0` rather than `0.1.0`.

## Overwrite a schema

Overwriting a schema (also called “patching” — not to be confused with the “patch” component of the version) is available if your schema version isn't yet in production.

> **Danger:** For Snowplow CDI customers, patching is disabled for production pipelines.
>
> Patching in a production environment can break your loading, especially if your patch contains breaking changes.
>
> Also, never patch a schema version that exists in a production environment, even if you are doing the patching in a development environment. This will lead to problems later when you try to promote that schema to production.

If you are working on a new schema version in a development environment, patching is safer as it won't corrupt any production data.

Before:

```mermaid
flowchart LR
  style v102 fill:orange
  %% ~~~ is not working for some reason
  linkStyle default stroke:none,fill:none
  v100("1-0-0") --- v101("1-0-1") --- v102("1-0-2\n(incorrect)")
```

After:

```mermaid
flowchart LR
  style v102 fill:teal
  linkStyle default stroke:none,fill:none
  v100("1-0-0") --- v101("1-0-1") --- v102("1-0-2\n(corrected)")
```

Follow these [instructions](/docs/event-studio/data-structures/versioning/#patch-a-schema) to patch a schema.

## Mark a schema as superseded

If your events are failing in production because of an incorrect schema, you might not be able to instantly update the tracking code to use a new schema version. This is a common situation for mobile tracking, for example. You can resolve this by marking the old schema version as superseded by the new schema version.

> **Note:** You need to be on Enrich 3.8.0+ and Iglu Server 0.11.0+ to use this feature. Additionally, if you are using [Snowplow Mini](/docs/api-reference/snowplow-mini/) or [Snowplow Micro](/docs/testing/snowplow-micro/), you will need version 0.17.0+ or 1.7.1+ respectively.

Before:

```mermaid
flowchart RL
  style v102 fill:orange
  linkStyle default stroke:none,fill:none
  v102("1-0-2\n(incorrect)") --- v101("1-0-1") --- v100("1-0-0")
```

After:

```mermaid
flowchart RL
  style v102 fill:orange
  style v103 fill:teal
  v102("1-0-2\n(incorrect)") --- v101("1-0-1") --- v100("1-0-0")
  v103("1-0-3\n(corrected)") -->|supersedes| v102
  linkStyle 0,1 stroke:none,fill:none
```

Here's how this works, at a glance:

- Suppose schema `1-0-2` is wrong.
- Draft a new schema version correcting the issue.
- In the new schema, add the following field at the root: `"$supersedes": ["1-0-2"]`.
- [Set the version](/docs/event-studio/data-structures/versioning/) of the new schema as usual, i.e. `1-0-3` if there are no breaking changes or `2-0-0` if there are.
- [Add](/docs/event-studio/data-structures/) the new schema to your production environment.
- Events or entities that use schema `1-0-2` will now be automatically updated (in the Enrich application) to use version `1-0-3`, and will be validated against that version. (A special entity will be added to these events to record this fact.)

### Example

Let’s say we have a mobile application. We are sending certain events from this application, and these events contain entities with following schema:

**Geolocation 1-0-2**

```json
{

    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

    "description": "Schema for client geolocation contexts",

    "self": {

        "vendor": "com.acme",

        "name": "geolocation",

        "format": "jsonschema",

        "version": "1-0-2"

    },

    "type": "object",

    "properties": {

        "latitude": {

            "type": "number",

        },

        "longitude": {

            "type": "number",

        }

    },

    "additionalProperties": false

}
```

Later, we realize that when implementing tracking, we have mistakenly included an `altitude` field in the entity objects:

**Wrong tracking code (iOS)**

```swift
let event = ScreenView(name: "Screen")

event.entities.add(

    SelfDescribingJson(schema: "iglu:com.acme/geolocation/jsonschema/1-0-2",

        andDictionary: [

            "latitude": 38.7223,

            "longitude": 9.1393,

            "altitude": 20 // extra field not defined in the schema

        ])!)

tracker.track(event)
```

Since `additionalProperties` is set to `false`, all events with the `altitude` field end up as [failed events](/docs/fundamentals/failed-events/).

We can create a new schema with version `1-0-3` that contains the `altitude` field and then use this schema in the next version of the application. This would make the events valid. However, users will not update their application to the new version all at once. Events from the older version will continue to come, therefore there will still be failed events until all users start to use a newer version.

To solve this problem, we simply add the `$supersedes` definition to the new schema.

**Geolocation 1-0-3 with $supersedes**

```json
{

    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

    "$supersedes": ["1-0-2"],

    "description": "Schema for client geolocation contexts",

    "self": {

        "vendor": "com.acme",

        "name": "geolocation",

        "format": "jsonschema",

        "version": "1-0-3"

    },

    "type": "object",

    "properties": {

        "latitude": {

            "type": "number",

        },

        "longitude": {

            "type": "number",

        },

        "altitude": {

            "type": "number",

        }

    },

    "additionalProperties": false

}
```

Now, when we receive events from the mobile application that use schema `1-0-2`, these events will be updated to use schema `1-0-3` and will be validated against that schema. Therefore, these events will be valid.

To record this fact, an extra entity will be added to all such events:

```json
{

    "schema": "iglu:com.snowplowanalytics.iglu/validation_info/jsonschema/1-0-0",

    "data": {

        "originalSchema": "iglu:com.acme/geolocation/jsonschema/1-0-2",

        "validatedWith": "1-0-3"

    }

}
```

Finally, if we [browse](/docs/event-studio/data-structures/) schema version `1-0-2`, we will see that Iglu Server automatically keeps track of which schema supersedes which. Specifically, it will now contain a `$supersededBy` definition:

**Geolocation 1-0-2 with $supersededBy**

```json
{

    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

    "$supersededBy": "1-0-3",

    "description": "Schema for client geolocation contexts",

    "self": {

        "vendor": "com.acme",

        "name": "geolocation",

        "format": "jsonschema",

        "version": "1-0-2"

    },

    "type": "object",

    "properties": {

        "latitude": {

            "type": "number",

        },

        "longitude": {

            "type": "number",

        }

    },

    "additionalProperties": false

}
```

### Usage

The `$supersedes` field states that the schema version defined in the `self` part supersedes the schema versions listed in the `$supersedes` field (one or more). Its value must be an array of strings (even if it only includes one item). For example:

```json
...

"$supersedes": ["1-0-2", "1-0-3"],

...
```

> **Note:** Once you’ve defined the `$supersedes` field for a schema version, you can’t update it — even in the development environment where patching is allowed. Likewise, you can’t patch (overwrite) an existing schema version to add `$supersedes`.
>
> You can only change which schema version supersedes which by creating _new_ schema versions.
>
> For example, if version `1-0-2` is defined to supersede version `1-0-1`, and you create version `1-0-3` which also supersedes `1-0-1`, then `1-0-1` will be superseded by the newest version, i.e. `1-0-3`. See diagrams below for more information on how this is determined.

> **Tip:** Note that various pipeline components, most importantly Enrich (including Enrich embedded in Snowplow Mini and Snowplow Micro), cache schemas to improve performance. The default caching time is 10 minutes (it’s controlled by the [Iglu Resolver configuration](/docs/api-reference/iglu/iglu-resolver/)). This means that the effect of superseding a schema will not be immediate.

### Rules

#### A schema version can only supersede previous versions

For example, `1-0-2` can supersede `1-0-1`, but _can’t_ supersede `1-0-3`, `1-1-0`, or `2-0-0`. Iglu Server will reject a schema with a definition that breaks this rule.

|                                    ✅ OK                                   |                                 ❌ Invalid                                 |
| :-----------------------------------------------------------------------: | :-----------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v102("1-0-2") -->|supersedes| v101("1-0-1")
``` | ```mermaid
flowchart LR
  v102("1-0-2") -->|supersedes| v200("2-0-0")
``` |

#### A schema version can supersede multiple previous versions at once

Events referencing either of those previous versions will be treated as explained above.

|                                                                                ✅ OK                                                                                |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v102("1-0-2") --- v101("1-0-1")
  v103("1-0-3") -->|supersedes| v101
  v103 -->|supersedes| v102
  linkStyle 0 stroke:none,fill:none
``` |

#### At any given moment, a schema version can only be superseded by a single schema version

Iglu Server automatically upholds this rule.

For example, if you specify that `1-0-3` supersedes `1-0-2` and (later) that `1-0-4` also supersedes `1-0-2`, the latest schema — `1-0-4` — will automatically become the one that supersedes `1-0-2`.

|                                                    Specified                                                    |                                                                      Becomes                                                                      |
| :-------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v103("1-0-3") -->|supersedes| v102("1-0-2")
  v104("1-0-4") --->|supersedes| v102
``` | ```mermaid
flowchart RL
  v104("1-0-4") --- v103("1-0-3") --- v102("1-0-2")
  v104 -->|supersedes| v102
  linkStyle 0,1 stroke:none,fill:none
``` |

The same happens if you specify “chains”, e.g. `1-0-3` supersedes `1-0-2` and `1-0-4` supersedes `1-0-3`. This will be automatically updated so that `1-0-4` supersedes `1-0-2` and `1-0-3`.

|                                                    Specified                                                   |                                                                               Becomes                                                                              |
| :------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v103("1-0-3") -->|supersedes| v102("1-0-2")
  v104("1-0-4") -->|supersedes| v103
``` | ```mermaid
flowchart RL
  v103("1-0-3") --- v102("1-0-2")
  v104("1-0-4") -->|supersedes| v102
  v104 -->|supersedes| v103
  linkStyle 0 stroke:none,fill:none
``` |

---

# Introduction to tracking design
> Learn how to design effective behavioral data tracking by analyzing use cases, defining entities and events, and creating comprehensive tracking plans.
> Source: https://docs.snowplow.io/docs/fundamentals/tracking-design-best-practice/

Good tracking design is essential for successful data collection. This guide will help you approach the design process systematically.

To use Snowplow successfully, you need to have a good idea of:

- What events you care about in your business
- What events occur in your website, mobile application, or server-side systems
- What decisions you make based on those events
- What you need to know about those events to make those decisions

The final outcome of your planning will be a set of tracking plans.

## Snowplow tracking concepts

![Tracking plan overview showing the relationship between tracking plans, event specifications, data structures](/assets/images/tracking-plan-overview-9f48fdeef0c5d4c593b416f9313c174c.png)

This diagram shows how tracking plans are conceptualized in Snowplow.

- **Tracking plans** are containers for related event specifications.
- **Event specification** represents a single business event. It contains all the relevant information about the event, including its purpose, origin, and associated data structures. Each has a single event data structure to define the event's properties, and can have multiple associated entity data structures.
- **Event and entity data structures** define the structure of the captured data to allow in JSON schemas for consistent data collection and analysis.

> **Info:** Snowplow customers can create [tracking plans](/docs/event-studio/tracking-plans/) and [event specifications](/docs/event-studio/tracking-plans/event-specifications/) directly in [Snowplow Console](https://console.snowplowanalytics.com).

For example, an `Ecommerce Checkout Flow` tracking plan may contain three event specifications:

- `Checkout Started` with a `checkout_started` event data structure and associated `cart` entity
- `Product Add To Cart` with an `add_to_cart` event data structure and associated `cart` and `product` entities
- `Order Completed` with an `order_completed` event data structure and associated `cart`, `product`, `order` and `payment` entities

Note that the `cart` entity data structure is reused across event specifications, this promotes consistency in your tracking design and makes analysis easier.

Additionally you can assign entities to [source applications](/docs/event-studio/source-applications/) to document which entities are expected for each event within that application, these are also called [global entities](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/). An example would be assigning a `user` entity to a mobile application to ensure that user information is always captured with events from that application.

## Naming conventions for tracking plans

A common standard for naming and structuring events, entities, and properties makes it easier to understand and extend your tracking design.

Here are our recommendations, but it's more important to be consistent across your tracking plans than to follow these guidelines:

For tracking plans:

- Use a descriptive name in title case that reflects the business domain, application, or use-case, e.g., `Ecommerce Checkout Flow`, `Mobile App User Engagement`, `SaaS Application Usage`

For Event Specifications:

- Use a verb-noun format in title case that clearly describes the action, e.g., `Add To Cart`, `User Signup`, `Purchase Completed`
- Be consistent with tense (e.g., all present or all past tense)
- Ensure each event specification represents a single event that has a clear purpose

For Event and Entity Data Structures:

- Use snake\_case for names, e.g., `add_to_cart`, `user_signup`, `purchase_completed`
- Use clear, descriptive names that reflect the purpose of the data structure

For Properties:

- Use snake\_case for property names, e.g., `user_id`, `product_name`,
- Be specific and descriptive to avoid ambiguity, e.g., use `purchase_amount` instead of just `amount`
- Do not repeat information contained in the data structure name, e.g., avoid `order_id` in an `order` entity

## Tracking plan best practices

Tracking plans are logical groupings of related business events with defined ownership. They help organize your tracking design and make it easier to manage and analyze your data.

When defining tracking plans, consider who will own the data and what business domain or use-case the events relate to. If you have multiple teams or departments, it can be helpful to align tracking plans with those organizational structures. Additionally, those teams can reflect implementation ownership or analysis ownership.

Examples of tracking plans include:

- `Ecommerce Checkout Flow`: contains events related to the checkout process in an ecommerce application
- `Mobile App User Engagement`: contains events related to user interactions within a mobile application
- `SaaS Application Usage`: contains events related to user actions within a SaaS platform

Bad examples of tracking plans would be overly broad or vague groupings, such as `All User Events` or `Miscellaneous Events`, which do not provide clear context or ownership. Another example would be overly specific groupings that limit reusability, such as `Product Page Views for Campaign X`.

When defining tracking plans, consider the following best practices:

- **Clear purpose**: each tracking plan should have a well-defined purpose and scope
- **Ownership**: assign clear ownership to each tracking plan to ensure accountability for data quality and governance
- **Logical grouping**: group related events that share a common business domain or use-case. Consider a group that reflects how the data will be used in analysis.
- **Reusability**: design tracking plans to promote reusability of event and entity data structures across different tracking plans

## Event Specification best practices

Event Specifications represent the key business events you are tracking. They contain a name, description, trigger conditions, and associated data structures. Event Specifications are designed to represent a single event to be implemented and analyzed. Each Event Specification should have one primary purpose.

For example, an `Add To Cart` Event Specification would represent the action of a user adding a product to their shopping cart. It would include:

- A clear name: `Add To Cart`
- A description of the event's purpose: "tracks when a user adds a product to their shopping cart"
- Trigger conditions: "fired when the user clicks the 'Add to Cart' button on a product page"
- Associated data structures: an `add_to_cart` event data structure and associated `product` and `cart` entity data structures

A bad example would be an `Ecommerce Action` Event Specification that tries to capture multiple actions like adding to cart, starting checkout, and completing a purchase in a single event. This can lead to confusion in implementation and complexity in analysis.

## Entity design best practices

**It is recommended that you adopt an "entity-first" approach to design**. This means starting by defining the key entities in your business domain before defining the events that interact with those entities. This approach helps ensure consistency and reusability across your tracking design.

If a piece of information is likely to be relevant to multiple different events, it belongs in an entity, not as a property of a single event. In fact, we often recommend not including any properties directly on event data structures, and instead placing all information in entities unless truly necessary.

Examples of common entities include:

- `user`: information about the user performing the action
- `product`: details about a product being viewed or purchased
- `cart`: information about a shopping cart
- `order`: details about an order being placed

## Event data structure granularity

A common challenge in defining event schemas is the choice of their granularity. We see customers struggle with this decision frequently in their tracking design.

### Approach 1: Group multiple actions into a single event schema

In some cases, it may be beneficial to group related actions into a single event schema. This means that a single event schema captures multiple types of actions, often distinguished by a property within the schema.

For example, you might define a single `ecommerce_action` event schema that includes a `type` property to distinguish between `view_product`, `add_to_cart`, `checkout_started`, and `purchase_completed` actions. Another example could be a `user_interaction` event schema that captures various user actions like `click`, `scroll`, and `form_submit`, with a `interaction_type` property to differentiate them.

This approach can be useful when:

- **Analysis**: the actions are closely related and often analyzed together
- **Simplicity**: you want to reduce the number of event schemas and columns in your data warehouse

Continuing the example from above, it is important to ensure the correct `type` property is set for each action and the allowed values are enforced through strong governance principles. This can be managed in Snowplow through Event Specifications with [property instructions](/docs/event-studio/tracking-plans/event-specifications/#properties). Tools like [Snowtype](/docs/event-studio/implement-tracking/) can also help simplify this complexity during implementation.

### Approach 2: One event schema per action

Defining a separate event schema for each action is often the most straightforward approach. This means that each event schema corresponds to a single user action or system event.

For example, in an ecommerce application, you might have separate event schemas for:

- `view_product`
- `add_to_cart`
- `checkout_started`
- `purchase_completed`

This approach has several advantages:

- **Clarity**: each event schema has a clear purpose, making it easier to understand
- **Flexibility**: you can evolve each event schema independently, without affecting others

However, this approach may lead to a large number of event schemas if your application has many distinct actions and can make analysis more complex.

---

# Introduction to tracking plans
> Snowplow's tracking plans enable organizations to easily generate AI and BI-ready data that is reliable, clear, compliant, accurate, and predictable
> Source: https://docs.snowplow.io/docs/fundamentals/tracking-plans/

Tracking plans are Snowplow's solution to helping organizations more easily create and democratise behavioral data. By creating well-documented datasets, that are both human and machine readable, you can more easily collaborate around data and unlock self-service analytics at scale.

With tracking plans, you can:

- Set clear ownership for the data being created
- Make tracking implementation easier
- Deliver better governance around your data
- More easily communicate what the data means and how to use it
- Collaborate more effectively with the various teams involved in delivering value from your data
- Drive a self-serve culture around data across your organization

## What is a Tracking Plan

A tracking plan is a documented dataset. By documenting what data you are collecting, where, the meaning of that data and how to use it, you can break down the barriers that often exist between the many teams involved in the data value chain, i.e. from those implementing the tracking to those analysing the data.

At its core, a tracking plan has:

- An explicit owner; that is responsible for maintaining and updating the data over time
- Consumers; who use the data to deliver use cases and are impacted by upstream changes

Tracking plans at Snowplow are underpinned by the concept of a data contract. They act as a formal agreement between the producers of tracking plans and the consumers of tracking plans, and support better collaboration around the data being created.

Examples of tracking plans:

- Ecommerce Web
- Media Web

## Key elements of a Tracking Plan

**The Source Application/s it is part of**; a tracking plan is referencing the [Source Application/s](/docs/event-studio/source-applications/) that is spanning across.

![Snowplow Console showing the \&quot;E-commerce Web\&quot; tracking plan Overview tab, including the tracking plan name, description, source application (website), ownership details, and a table of event specifications with their data structures, entities, source applications, last seen dates, and event volume counts.](/assets/images/example_tracking_plan_view-949245f8b596b73d8d03038828a465fd.png)

**Benefits:**

- Have a clear view in which application the tracking plan is implemented in, which domains it spans and the related application context information it will have available by default in the dataset.

**An owner**; tracking plans are typically split by domain with each tracking plan having an explicit owner that is responsible for the maintenance and evolution of that data.

**Benefits:**

- Know who to go to when new data is required for a particular domain or when there is an issue with the data, breaking down the barriers between data producers and data consumers

**Event specifications**; these describe each event that is collected as part of the tracking plan, on which applications they are triggered and where, the event data structure to validate against, and the entities to attach to each event (e.g. user, product etc).

![Snowplow Console showing the \&quot;Add to cart\&quot; event specification detail page, including event description, source application (website), application IDs with status indicators, the snowplow\_ecommerce\_action 1-0-2 data structure with a required \&quot;type\&quot; property set to the exact value \&quot;add\_to\_cart\&quot;, and entity data structures for product and cart.](/assets/images/example_event_specification-e0b48ec28bbc53327add421f607ff25f.png)

**Benefits:**

- Provide implementation details to developers implementing tracking (see section on [Snowtype](/docs/event-studio/implement-tracking/) for further details)
- Provide documentation around the semantics of the data that you are creating, to enable analysts, data scientists, analytics engineers with data discoverability

**Subscribers;** allow colleagues to express an interest in understanding changes that are made to the data within a Tracking Plan, usually because the data is being used in a downstream data model or Data Model Pack.

**Benefits:**

Break down the barriers that exist between data producers and data consumers, by providing:

- data producers, with visibility of how the data is being used downstream and the implications of any changes
- data consumers, with notifications when changes are made to a Tracking Plan so that they can make any necessary changes to their models, reports etc.

**Change history**; a complete audit log of the changes that have been made to the tracking plan, including changes to existing data and new data that has been added over time.

**Benefits:**

- Enhances accountability and transparency by providing a clear audit trail of all data modifications, fostering confidence in data integrity

**Volume metrics**; tracking plans can detect events ingested in your pipeline that match the configured event specifications. This allows your team to monitor occurrence-related metrics for events being tracked with specific event specification IDs.

**Benefits:**

You will be able to view several items in the UI that help detect anomalies or potential misconfigurations in trackers that are either not sending the expected events or are using incorrect [application IDs](/docs/event-studio/source-applications/#application-ids). This is particularly useful during the development phase when implementing tracking for a specific application using [Snowtype](/docs/event-studio/implement-tracking/). These elements include:

- A counter for each event specification, showing the total number of events detected from the tracked application IDs in the last 30 days.

- A 'last seen' field for each event specification, indicating when the last event matching the event specification ID was detected.

- A list of application IDs from which events are being tracked, displayed for each event specification. For each application ID, a status will be shown with different colors:

  - **Green**: Event specifications are being tracked and identified with the specific application ID inherited from the configured [source applications](/docs/event-studio/source-applications/).
  - **Gray**: No event specifications are being tracked for an application ID inherited from the configured source applications.
  - **Yellow**: Event specifications are being tracked for an application ID that has not been configured or inherited from the source applications.

> **Note:** Some tracking plans, such as [**Base Web**](/docs/event-studio/tracking-plans/templates/#base-web) and [**Base Mobile**](/docs/event-studio/tracking-plans/templates/#base-mobile), contain standard events (e.g., _page pings_, _link clicks_, _screen view_, _application install_).
>
> For these tracking plans, the volume metrics will behave differently:
>
> - If no standard events are being tracked with an application ID different from those inherited from the source applications set up in the tracking plan, the behavior will be the same as for a normal tracking plan.
>
> - If standard events are being tracked with application IDs different from those inherited from the source applications set up in the tracking plan, a toggle will appear above the event specification list.
>
>   This toggle will be disabled by default, so the metrics displayed will relate only to the application IDs inherited from the source applications set up in the tracking plan.
>
>   ![Event specifications list with the \&quot;Show all tracked App IDs\&quot; toggle disabled. Five event specifications are shown (Button click, Custom event, Link click, Page ping, Page view) with their data structures, the console-qa application ID badge, last seen dates, and volume counts. Page view shows 64.00k events.](/assets/images/tracking_plan_metrics_default-4ce71a9e5107ab9668dada2032937b6e.png)
>
>   If the toggle is enabled, it will show the metrics for all the application IDs found for the standard events (not just the ones inherited from the tracking plan).
>
>   ![Event specifications list with the \&quot;Show all tracked App IDs\&quot; toggle enabled. The Page view event now shows three application ID badges — console-qa (green indicator), react-app (yellow indicator), and duncan\_app (yellow indicator) — revealing events tracked from application IDs not configured in the tracking plan\&#39;s source applications.](/assets/images/tracking_plan_metrics_toggled-5f17d4d14eaf50490650ceecf5e47791.png)

## How tracking plans help with governance, data quality and data discoverability

The **data structures** that you attach to your event specification describe the [structure of the data](/docs/fundamentals/schemas/). They validate that the values of the properties contained within your events and entities are **valid** as they pass through your pipeline.

The **event specification** describes a specific implementation of an event. It is a narrower definition of your event than a data structure - not only do they describe the structure of the data (by attaching the relevant data structure to validate against), they also allow you to define the **right values** for fields when the event gets triggered and the entities that need to be attached.

_NB: the event specifications are not enforced by your pipeline (for example, we don’t yet validate that the correct entities are attached to an event)._

By adding screenshots, and descriptions to the event specification, you are also able to communicate the **semantics of the data** to those that want to analyse it. In this way you can ensure that the data is **represented accurately** when being used to derive insights and make decisions by the many teams using the data downstream.

A data structure can be used across event specifications, and across tracking plans. In doing so, you can ensure you consistently track business critical events and entities (for example, your "product" entity) across your organisation. Having the ability to use centralised event and entity schemas in this way, means that you are able to better govern the structure of the data across an organisation whilst also empowering teams to manage their own specific implementation of events via tracking plans.

To understand how to get started with tracking plans, see [Defining the Data to collect with Tracking Plans](/docs/event-studio/tracking-plans/) for further details.

---

# Introduction to the atomic events table
> A summary of the Snowplow events table and its fields, including custom events and entities
> Source: https://docs.snowplow.io/docs/fundamentals/warehouse-tables/

All Snowplow events have the same underlying structure and standard fields. All of these fields can be found in the `atomic.events` table, which is a "wide" (many columns) table.

Each line in the atomic events table represents a single event, be that a `page_view`, `add_to_basket`, `play_video`, etc.

Individual fields are stored in their own columns. Check out the [event properties reference](/docs/fundamentals/canonical-event/) for a full list of standard fields. [Self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) are stored as additional columns in the `atomic.events` table (except in Redshift, see below).

> **Tip:** The Snowplow data table is designed to be immutable: the data in each line should not change over time.
>
> Data points that you would expect to change over time e.g. what cohort a particular user belongs to, or how you classify a particular visitor, can be derived from Snowplow data. Our recommendation is that you define and calculate these derived fields at analysis time, store them in a separate table, and join to the `atomic.events` table when performing any analysis.

Learn about querying Snowplow data [here](/docs/destinations/warehouses-lakes/querying-data/).

## Redshift table structure

In Redshift, unlike other warehouses, [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) are stored in their own dedicated tables.

These additional tables can be joined back to the core `atomic.events` table, by joining on the `root_id` field in the self-describing event or entity table with the `event_id` in the `atomic.events` table, and the `root_tstamp` and `collector_tstamp` field in the respective tables.

You can still query the data as if it were in a single wide table. This is because:

- The joins from the additional tables to the core `atomic.events` table are one-to-one
- The field joined on is the distribution key for both tables, so queries are as fast as if the data were in a single table

## Schemas in the warehouse

To understand how self-describing events and entities are translated into warehouse columns, check out this [reference page](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

---

# Get started with Console
> Use the Getting Started dashboard in Console to design tracking, implement it, deploy data models, and configure Signals.
> Source: https://docs.snowplow.io/docs/get-started/console/

The Getting Started dashboard in Console guides you through collecting and acting on your behavioral data. It organizes workflows into three sections: **Fundamentals**, **Analytics**, and **Real-time use cases**.

![The Getting Started dashboard in Console showing Fundamentals, Analytics, and Real-time use cases sections with guided workflow cards](/assets/images/console-getting-started-055d1e21167901b1ef5a52e76ab12cf8.png)

## How the dashboard is organized

Each section contains workflow cards for a specific task, from designing a tracking plan to configuring Signals. The workflows follow a prerequisite chain: create a tracking plan, implement tracking, then unlock downstream features like data models and Signals.

> **Note:** The Getting Started dashboard is available to Snowplow CDI customers with Console access.

## Design and implement tracking

The Fundamentals section covers the core data collection workflow.

### Create a tracking plan

The first card guides you through creating a [tracking plan](/docs/event-studio/tracking-plans/), which defines the [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/) you want to collect.

1. Choose between built-in events or custom tracking
2. Create or select a [source application](/docs/event-studio/source-applications/)
3. Select tracking plan templates (Base Web, Ecommerce Web, Media Web, and others)
4. Review and confirm your selections

Console then creates the tracking plan and associated data structures in your account.

### Implement tracking

With a tracking plan in place, the next card helps you [implement tracking](/docs/event-studio/implement-tracking/).

1. Select your source application
2. Configure tracker initialization with the generated code snippet
3. Select your tracking plan and view event-specific code snippets
4. Test with Snowplow Micro
5. Deploy to production

> **Tip:** Test your implementation with Snowplow Micro before deploying to production to verify that events are structured correctly and match your tracking plan.

## Deploy a data model

Once data is flowing, you can deploy a data model to transform raw events into structured tables for analysis. Console offers two types of data models:

- [Out-of-the-box data models](/docs/modeling-your-data/running-data-models-via-console/) for common use cases like web and mobile analytics
- [Automatically generated data models](/docs/modeling-your-data/automatically-generated-data-models/) tailored to your tracking plan

## Configure Signals for real-time use cases

[Signals](/docs/signals/get-started/) processes your event stream to compute user attributes in real time and trigger automated actions.

> **Note:** Signals is a paid add-on. Contact your account team to enable it.

### Design Signals attributes

Define [Signals attributes](/docs/signals/attributes/) that compute real-time properties about your users (for example, session count, cart value, or engagement score).

1. Choose a template or create a custom attribute
2. Create or edit an attribute group
3. Test and simulate against real data

### Retrieve Signals attributes

Create a [service](/docs/signals/applications/services/) and follow the [retrieval steps](/docs/signals/applications/retrieve-attributes/) to access attribute values from your front-end or back-end.

### Trigger interventions

Create [interventions](/docs/signals/interventions/) that automatically respond to user behavior — for example, displaying a promotion when a cart value exceeds a threshold or suppressing a campaign for users who have already converted.

---

# Creating your first custom events
> How to define your own events and entities
> Source: https://docs.snowplow.io/docs/get-started/custom-events/

While Snowplow provides various [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/) out of the box, you can define your own to match your business more closely.

The documentation on [self-describing events](/docs/fundamentals/events/#self-describing-events) provides a high-level view of how you might create and track one. For more guidance, check out [this tutorial](/tutorials/data-products-base-tracking/start/).

---

# Inspecting your first failed events
> How validation and failed events work
> Source: https://docs.snowplow.io/docs/get-started/failed-events/

If you have defined some custom events in the previous section, we recommend playing around with sending invalid data to see what happens.

For example, if your custom event has a numeric field, try sending a string. It will not reach your warehouse! Instead, a [failed event](/docs/fundamentals/failed-events/) will be generated.

> **Tip:** If you have not defined any custom events yet, you can use the following command-line snippet to send yourself an invalid event (it uses a nonexistent schema `com.my_company/product_view/jsonschema/1-0-0`):
>
> ```bash
> curl 'https://{{COLLECTOR_URL}}/com.snowplowanalytics.snowplow/tp2' \
>
>    -H 'Content-Type: application/json; charset=UTF-8' \
>
>    -H 'Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c' \
>
>    --data-raw '{"schema":"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4","data":[{"e":"ue","ue_px":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3Byb2R1Y3Rfdmlldy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJpZCI6IjVOMFctUEwwVyIsImN1cnJlbnRfcHJpY2UiOjQ0Ljk5LCJkZXNjcmlwdGlvbiI6IlB1cnBsZSBTbm93cGxvdyBIb29kaWUifX19","tv":"js-2.17.2","tna":"spExample","aid":"docs-example","p":"web","tz":"Europe/London","lang":"en-GB","cs":"UTF-8","res":"3440x1440","cd":"24","cookie":"1","eid":"542a79d3-a3b8-421c-99d6-543ff140a56a","dtm":"1626182778193","cx":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiI0YTU2ZjQyNy05MTk2LTQyZDEtOWE0YS03ZjRlNzk2OTM3ZmEifX1dfQ","vp":"863x1299","ds":"848x5315","vid":"3","sid":"87c18fc8-2055-4ec4-8ad6-fff64081c2f3","duid":"5f06dbb0-a893-472b-b61a-7844032ab3d6","refr":"https://docs.snowplow.io/","url":"/docs/","stm":"1626182778194"}]}'
> ```

Follow the [documentation](/docs/monitoring/exploring-failed-events/) to explore your failed events.

---

# Product features in Snowplow CDI and Self-Hosted
> Detailed comparison of features available in Snowplow CDI and Self-Hosted deployments.
> Source: https://docs.snowplow.io/docs/get-started/feature-comparison/

Here is a detailed list of product features, showing which are available as part of Snowplow [Customer Data Infrastructure](/docs/get-started/#customer-data-infrastructure) (CDI) or [Snowplow Self-Hosted](/docs/get-started/#self-hosted).

Check out the [Snowplow Product Directory](https://snowplow.io/snowplow-product-description) to learn more about which features are available for CDI Cloud or CDI Private Managed Cloud deployments.

| ### Data Pipeline                                                                                                    | CDI | Self-Hosted      |
| -------------------------------------------------------------------------------------------------------------------- | --- | ---------------- |
| [25+ trackers and webhooks](/docs/sources/)                                                                          | ✅   | ✅                |
| First party tracking                                                                                                 | ✅   | ✅                |
| Anonymous data collection                                                                                            | ✅   | ✅                |
| [Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/)          | ✅   | ✅                |
| High availability and auto-scaling                                                                                   | ✅   | ❌                |
| [Enrichments](/docs/pipeline/enrichments/available-enrichments/)                                                     | ✅   | ✅                |
| [Failed events](/docs/fundamentals/failed-events/)                                                                   | ✅   | ✅                |
| [Data quality monitoring](/docs/monitoring/)                                                                         | ✅   | ❌                |
| Single Sign-On                                                                                                       | ✅   | ❌                |
| Pipeline observability                                                                                               | ✅   | do-it-yourself   |
| Surge protection                                                                                                     | ✅   | do-it-yourself   |
| **Warehouse / lake destinations**                                                                                    |     |                  |
| • Snowflake                                                                                                          | ✅   | ✅                |
| • Redshift                                                                                                           | ✅   | ✅                |
| • BigQuery                                                                                                           | ✅   | ✅                |
| • Databricks                                                                                                         | ✅   | ✅                |
| • Synapse Analytics                                                                                                  | ✅   | ✅                |
| • Elasticsearch                                                                                                      | ✅   | ✅                |
| • S3                                                                                                                 | ✅   | ✅                |
| • Google Cloud Storage                                                                                               | ✅   | ✅                |
| • Azure Data Lake Storage / OneLake                                                                                  | ✅   | ✅                |
| **Real-time streams**                                                                                                |     |                  |
| • Kinesis                                                                                                            | ✅   | ✅                |
| • PubSub                                                                                                             | ✅   | ✅                |
| • Kafka / Azure Event Hubs / Confluent Cloud                                                                         | ✅   | ✅                |
| ### Event Studio                                                                                                     | CDI | Self-Hosted      |
| Advanced enrichments (PII, IP anonymization, JS, API, SQL enrichments)                                               | ✅   | ✅ (no UI or API) |
| [Data structures tooling and API](/docs/event-studio/data-structures/)                                               | ✅   | ❌                |
| [Tracking plans](/docs/event-studio/tracking-plans/)                                                                 | ✅   | ❌                |
| [Data modeling management tooling](/docs/modeling-your-data/running-data-models-via-console/)                        | ✅   | ❌                |
| Jobs monitoring dashboard                                                                                            | ✅   | ❌                |
| Failed events alerting                                                                                               | ✅   | ❌                |
| Failed events in the warehouse                                                                                       | ✅   | ✅                |
| QA pipeline                                                                                                          | ✅   | do-it-yourself   |
| Fine-grained user permissions using access control lists                                                             | ✅   | ❌                |
| API key access                                                                                                       | ✅   | ❌                |
| ### [Data Model Packs](/docs/modeling-your-data/visualization/)                                                      | CDI | Self-Hosted      |
| **Digital Analytics**                                                                                                |     |                  |
| Funnel builder                                                                                                       | ✅   | ❌                |
| User and Marketing Analytics                                                                                         | ✅   | ❌                |
| Marketing Attribution                                                                                                | ✅   | ❌                |
| Video and Media                                                                                                      | ✅   | ❌                |
| [Unified Digital, Attribution, Media Player, Ecommerce, Normalize, Utils data models](/docs/modeling-your-data/)     | ✅   | ❌                |
| **Ecommerce Analytics**                                                                                              |     |                  |
| Ecommerce Analytics                                                                                                  | ✅   | ❌                |
| [Ecommerce data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/)    | ✅   | ❌                |
| ### Signals                                                                                                          | CDI | Self-Hosted      |
| [Real-time personalization engine](/docs/signals/introduction/)                                                      | ✅   | ❌                |
| Profiles Store                                                                                                       | ✅   | ❌                |
| Interventions                                                                                                        | ✅   | ❌                |
| ### Identities                                                                                                       | CDI | Self-Hosted      |
| [Real-time identity resolution](/docs/identities/)                                                                   | ✅   | ❌                |
| [Identities dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-identities-data-model/) | ✅   | ❌                |
| ### Performance and Resilience                                                                                       | CDI | Self-Hosted      |
| Outage Protection                                                                                                    | ✅   | ❌                |
| Global Availability                                                                                                  | ✅   | ❌                |
| ### Infrastructure and Security                                                                                      | CDI | Self-Hosted      |
| **High**                                                                                                             |     |                  |
| HTTP access controls                                                                                                 | ✅   | ❌                |
| VPC peering                                                                                                          | ✅   | ❌                |
| SSH access control                                                                                                   | ✅   | ❌                |
| CVE reporting                                                                                                        | ✅   | ❌                |
| Static collector IPs                                                                                                 | ✅   | ❌                |
| **Advanced**                                                                                                         |     |                  |
| Custom VPC integration                                                                                               | ✅   | ❌                |
| Custom IAM policy                                                                                                    | ✅   | ❌                |
| Custom security agents                                                                                               | ✅   | ❌                |
| ### SLAs                                                                                                             | CDI | Self-Hosted      |
| Collector uptime SLA                                                                                                 | ✅   | ❌                |
| Warehouse loading latency SLA                                                                                        | ✅   | ❌                |

---

# Snowplow free trial
> Start a free 14-day trial of Snowplow CDI Cloud, and learn about the product and capacity limitations that apply during the trial.
> Source: https://docs.snowplow.io/docs/get-started/free-trial/

The Snowplow free trial is a 14-day trial of [Snowplow CDI Cloud](/docs/get-started/#cdi-cloud). Snowplow hosts and manages the pipeline infrastructure, so you can collect, validate, and load behavioral data without setting up any cloud infrastructure yourself.

Use the trial to evaluate Snowplow end to end: define your data, send events from your application, and load validated data into a warehouse or lake destination.

## Start the trial

Sign up for the free trial on the [Snowplow website](https://snowplow.io/get-started/snowplow-free-trial).

After you sign up, your trial environment is provisioned automatically. You will receive access instructions within around 20 minutes.

The trial runs for 14 days. To request an extension, [contact sales](https://console.snowplowanalytics.com/contact-us).

## Product limitations

Some Snowplow features are not available during the trial:

- First-party tracking using your own domain is not available.
- Some destinations are not available during the trial. [Contact sales](https://console.snowplowanalytics.com/contact-us) to discuss your requirements.

## Capacity limits

The trial includes a limited set of resources that do not scale:

- Destinations are limited to:

  - one [Micro](/docs/testing/snowplow-micro/) instance
  - one [warehouse or lake destination](/docs/destinations/warehouses-lakes/)
  - one [event forwarding destination](/docs/destinations/forwarding-events/)

- Collector-level rate limits apply, capped at around 20 events per second.

- Pipeline, [Signals](/docs/signals/), and [Identities](/docs/identities/) resources are limited and do not scale during the trial.

> **Note:** For the full set of features available in Snowplow CDI, see the [feature comparison](/docs/get-started/feature-comparison/).

---

# First steps with Snowplow
> Compare Snowplow deployment options including CDI Private Managed Cloud, CDI Cloud, Community Edition, and Self-Hosted to choose the right platform for your organization.
> Source: https://docs.snowplow.io/docs/get-started/

Snowplow is the customer context layer that provides the data foundation for your organization. It transforms raw behavioral data into real-time customer context for AI agents and advanced analytics, using [Snowplow Signals](/docs/signals/) and the Snowplow Customer Data Infrastructure (CDI) platform.

- Start by [defining the data](/docs/fundamentals/tracking-plans/) you want to collect
- Choose from over 20 [tracker SDKs](/docs/sources/) to collect data from web, mobile, server-side, and other sources, using custom tracking or [over 100 built-in events and entities](/docs/events/ootb-data/)
- Enrich your events before loading with over 15 [enrichments](/docs/pipeline/enrichments/available-enrichments/)
- Monitor your [data quality](/docs/fundamentals/failed-events/) - your pipeline will only load [validated data](/docs/fundamentals/schemas/) into your [warehouse or lake](/docs/destinations/warehouses-lakes/)
- Send enriched events to [third-party platforms](/docs/destinations/forwarding-events/) in real-time
- Model your data with our [dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/)
- Enable real time personalization with [Signals](/docs/signals/introduction/)

![diagram of snowplow architecture](/img/snowplow-cdi-signals_vertical.svg)

Choose the Snowplow platform that works for your business. See the [feature comparison page](/docs/get-started/feature-comparison/) for more information.

We offer two fully featured Customer Data Infrastructure (CDI) platforms:

- **Snowplow CDI Private Managed Cloud**: hosted in your own cloud, managed by Snowplow
- **Snowplow CDI Cloud**: hosted and managed by Snowplow

For self-hosted deployments, we have:

- **Snowplow Community Edition**: not for production use, hosted and managed by you
- **Snowplow Self-Hosted**: for production use, for existing Snowplow users

> **Tip:** To compare platforms before committing, start a [free 14-day trial](/docs/get-started/free-trial/) of CDI Cloud.

## Customer Data Infrastructure

Snowplow CDI is our full infrastructure offering. Choose whether you'd like the **data plane** to be entirely hosted in your cloud account, or whether you'd prefer Snowplow to host the pipeline infrastructure for you.

The **control plane**, which includes a UI and an API for [defining your data](/docs/event-studio/tracking-plans/) and managing your infrastructure, is always hosted by Snowplow.

### CDI Private Managed Cloud

Private Managed Cloud is a version of [Snowplow](https://snowplow.io) hosted in your own cloud account, using your data warehouse or lake. Ongoing pipeline maintenance, such as upgrades and security patches, are managed by Snowplow.

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | ✅                  |               |
| API endpoints                               | ✅                  |               |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) |                    | ✅             |
| Data destination (warehouse / lake)         |                    | ✅             |

### CDI Cloud

Cloud is a hosted version of Snowplow designed to get your organization up and running and delivering value from behavioral data as quickly as possible. With Cloud, you don't need to set up any cloud infrastructure yourself.

All data processed and collected with Snowplow Cloud is undertaken within Snowplow's own cloud account.

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | ✅                  |               |
| API endpoints                               | ✅                  |               |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) | ✅                  |               |
| Data destination (warehouse / lake)         |                    | ✅             |

## Self-Hosted

With Self-Hosted, you deploy and host everything. Many features, including the **control plane**, are not available in Self-Hosted.

### Community Edition

Snowplow [Community Edition](/docs/get-started/self-hosted/) is for **non-production** use cases. It's a starter template: use it to evaluate Snowplow for testing purposes.

Community Edition infrastructure is provided under the [SLULA license](/docs/licensing/).

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | N/A                | N/A           |
| API endpoints                               | N/A                | N/A           |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) |                    | ✅             |
| Data destination (warehouse / lake)         |                    | ✅             |

### Production Self-Hosted license

If you have an existing Snowplow implementation, either Community Edition or a legacy deployment, you're eligible for Snowplow Self-Hosted. It's a production-use license.

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | N/A                | N/A           |
| API endpoints                               | N/A                | N/A           |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) |                    | ✅             |
| Data destination (warehouse / lake)         |                    | ✅             |

---

# Running your first data models
> Using a data model to aggregate your data
> Source: https://docs.snowplow.io/docs/get-started/modeling/

Querying the events table directly — as you would have done in the previous step — can be useful for exploring your events or building custom analytics. However, for many common use cases it's much easier to use our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/), which provide a pre-aggregated view of your data.

We recommend [dbt](https://www.getdbt.com/) for data modeling. Refer to the [setup instructions](/docs/modeling-your-data/running-data-models-via-console/) to add and configure your models in [Snowplow Console](https://console.snowplowanalytics.com), so that they can be run automatically by Snowplow.

The [Unified Digital model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) is a good starting point for websites and/or mobile applications. It provides data about page and screen views, sessions, users, and more. You can also explore the [full list of available models](/docs/modeling-your-data/modeling-your-data-with-dbt/).

If you don't have access to Snowplow Console, you'll need to install dbt and run the models yourself. Check out our [Unified Digital Quick Start tutorial](/tutorials/unified-digital/intro/) for help getting started.

> **Tip:** To start using our models with dbt, you will need to [create a dbt project](https://docs.getdbt.com/reference/commands/init) and [add the respective packages](https://docs.getdbt.com/docs/build/packages).

---

# Setting up Snowplow CDI
> Get started with Snowplow Customer Data Infrastructure on Cloud or Private Managed Cloud with setup guides for AWS, Azure, and GCP.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/

To get started with Snowplow Customer Data Infrastructure, follow the **Getting Started** steps in [Snowplow Console](https://console.snowplowanalytics.com). You will receive an account as part of your onboarding.

## CDI Cloud

If you have a Snowplow [CDI Cloud](/docs/get-started/#cdi-cloud) account, we'll set up your infrastructure for you. Check out **Pipelines** in Console to see your new pipeline.

## CDI Private Managed Cloud

If you have a Snowplow [CDI Private Managed Cloud](/docs/get-started/#cdi-private-managed-cloud) account, the Getting Started steps will guide you through setting up your cloud environment. You can also find the instructions here:

- [AWS Setup Guide](/docs/get-started/private-managed-cloud/setup-guide-aws/)
- [Azure Setup Guide](/docs/get-started/private-managed-cloud/setup-guide-azure/)
- [GCP Setup Guide](/docs/get-started/private-managed-cloud/setup-guide-gcp/)

Once you've set up your cloud environment, go to **Pipelines** in Console to request your new pipeline.

---

# Private Managed Cloud on AWS
> Set up Snowplow CDI Private Managed Cloud on AWS with sub-account configuration, IAM roles, and cross-account access.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/setup-guide-aws/

To set up Snowplow, follow the ['Getting Started' steps in the Snowplow Console](https://console.snowplowanalytics.com). You will receive an account as part of your onboarding.

## What are the steps

The first setup steps are designed to get your infrastructure in place and have you sending data as quickly as possible. The initial steps include:

- providing the right cloud environment for Snowplow to be installed
- setting up your first development environment
- selecting which warehouse, if any, you want to load your data into

Completing the forms for these initial steps should take you around 30 minutes.

## What will I need

To set up your cloud environment as required you will need:

- to be able to set up a sub-account and appropriate permissions on AWS
- to know which AWS region you’d like us to install your Snowplow pipeline into
- to know whether or not you will need to use an existing VPC or require VPC peering (note: VPC peering and using a custom VPC are additional bolt-ons)

We often find our point of contact requires support from their DevOps or Networking colleagues to complete the cloud setup step; in Console you can [easily create accounts for colleagues](/docs/account-management/managing-users/) who can complete this step for you.

## Preparing your AWS sub-account

These instructions are also provided as part of the setup flow in Console.

### Create sub-account

1. From your main AWS account, set up an Organization if you haven't done so already.
2. Create a member account (the sub-account) in that organization.
3. Sign out and sign into the new sub-account. Everything Snowplow-related will take place within this account from here in.

### Set up Role and IAM permissions

> **Note:** Before you begin, raise the [Managed policies per role](https://us-east-1.console.aws.amazon.com/servicequotas/home/services/iam/quotas/L-0DA4ABF3) attached to an IAM role Service Quota in IAM (region: us-east-1) from the default of 10 to 25. This is typically auto-approved within seconds of requesting.

1. In the AWS sub-account, open IAM from the console
2. Navigate to `Access management` > `Roles` and click `Create role`
3. Under trusted entity type, select `AWS account`, then choose `Another AWS account`

- Account ID: `793733611312`
- Leave `Require MFA` unchecked — Snowplow uses Okta for MFA, which handles authentication before role assumption

4. Search for and attach each of the following AWS managed policies:

```text
arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess

arn:aws:iam::aws:policy/AmazonEC2FullAccess

arn:aws:iam::aws:policy/AmazonECS_FullAccess

arn:aws:iam::aws:policy/AmazonElastiCacheFullAccess

arn:aws:iam::aws:policy/AmazonEMRFullAccessPolicy_v2

arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess

arn:aws:iam::aws:policy/AmazonKinesisFullAccess

arn:aws:iam::aws:policy/AmazonMSKFullAccess

arn:aws:iam::aws:policy/AmazonOpenSearchServiceFullAccess

arn:aws:iam::aws:policy/AmazonRDSFullAccess

arn:aws:iam::aws:policy/AmazonRedshiftFullAccess

arn:aws:iam::aws:policy/AmazonRoute53FullAccess

arn:aws:iam::aws:policy/AmazonRoute53ResolverFullAccess

arn:aws:iam::aws:policy/AmazonS3FullAccess

arn:aws:iam::aws:policy/AmazonSNSFullAccess

arn:aws:iam::aws:policy/AmazonSQSFullAccess

arn:aws:iam::aws:policy/AmazonSSMFullAccess

arn:aws:iam::aws:policy/AutoScalingFullAccess

arn:aws:iam::aws:policy/AWSCertificateManagerFullAccess

arn:aws:iam::aws:policy/AWSLambda_FullAccess

arn:aws:iam::aws:policy/CloudWatchFullAccess

arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess

arn:aws:iam::aws:policy/IAMFullAccess
```

5. Name the role `SnowplowAdmin` — this exact name is required
6. Open the role you just created, go to the Permissions tab and click `Add permissions` > `Create inline policy`. Switch to the JSON editor, paste the following, and save the policy:

```json
{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Sid": "EKSClusterManagement",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "eks:AccessKubernetesApi",

        "eks:AssociateAccessPolicy",

        "eks:AssociateIdentityProviderConfig",

        "eks:CreateAccessEntry",

        "eks:CreateAddon",

        "eks:CreateCluster",

        "eks:CreateFargateProfile",

        "eks:CreateNodegroup",

        "eks:DeleteAccessEntry",

        "eks:DeleteAddon",

        "eks:DeleteCluster",

        "eks:DeleteFargateProfile",

        "eks:DeleteNodegroup",

        "eks:DeregisterCluster",

        "eks:DescribeAccessEntry",

        "eks:DescribeAddon",

        "eks:DescribeAddonConfiguration",

        "eks:DescribeAddonVersions",

        "eks:DescribeCluster",

        "eks:DescribeFargateProfile",

        "eks:DescribeIdentityProviderConfig",

        "eks:DescribeNodegroup",

        "eks:DescribeUpdate",

        "eks:DisassociateAccessPolicy",

        "eks:DisassociateIdentityProviderConfig",

        "eks:ListAccessEntries",

        "eks:ListAccessPolicies",

        "eks:ListAddons",

        "eks:ListAssociatedAccessPolicies",

        "eks:ListClusters",

        "eks:ListFargateProfiles",

        "eks:ListIdentityProviderConfigs",

        "eks:ListNodegroups",

        "eks:ListTagsForResource",

        "eks:ListUpdates",

        "eks:RegisterCluster",

        "eks:TagResource",

        "eks:UntagResource",

        "eks:UpdateAccessEntry",

        "eks:UpdateAddon",

        "eks:UpdateClusterConfig",

        "eks:UpdateClusterVersion",

        "eks:UpdateNodegroupConfig",

        "eks:UpdateNodegroupVersion"

      ]

    },

    {

      "Sid": "KMS",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "kms:DescribeKey",

        "kms:List*"

      ]

    },

    {

      "Sid": "SecretsManager",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "secretsmanager:CreateSecret",

        "secretsmanager:DeleteSecret",

        "secretsmanager:DescribeSecret",

        "secretsmanager:GetResourcePolicy",

        "secretsmanager:GetSecretValue",

        "secretsmanager:PutSecretValue",

        "secretsmanager:TagResource"

      ]

    },

    {

      "Sid": "GlobalAccelerator",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "globalaccelerator:AddEndpoints",

        "globalaccelerator:Create*",

        "globalaccelerator:Delete*",

        "globalaccelerator:Describe*",

        "globalaccelerator:List*",

        "globalaccelerator:TagResource",

        "globalaccelerator:UntagResource",

        "globalaccelerator:Update*"

      ]

    },

    {

      "Sid": "ServiceQuotas",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "servicequotas:Associate*",

        "servicequotas:Delete*",

        "servicequotas:Disassociate*",

        "servicequotas:Get*",

        "servicequotas:List*",

        "servicequotas:Put*",

        "servicequotas:Request*",

        "servicequotas:TagResource",

        "servicequotas:UntagResource"

      ]

    },

    {

      "Sid": "WAFv2ReadOnly",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "wafv2:Describe*",

        "wafv2:Get*",

        "wafv2:List*"

      ]

    },

    {

      "Sid": "Support",

      "Effect": "Allow",

      "Resource": "*",

      "Action": [

        "support:Add*",

        "support:Create*",

        "support:Describe*",

        "support:Refresh*",

        "support:ResolveCase"

      ]

    }

  ]

}
```

7. Go to the Trust relationships tab and click **Edit trust policy**. Replace the generated JSON with the following and click **Update policy**:

```json
{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "AWS": "arn:aws:iam::793733611312:root"

      },

      "Action": "sts:AssumeRole",

      "Condition": {

        "Bool": {

          "aws:MultiFactorAuthPresent": "false"

        },

        "StringLike": {

          "aws:PrincipalArn": "arn:aws:iam::793733611312:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_sre-*"

        }

      }

    }

  ]

}
```

You will need to share this role with us as part of filling out the setup form in Snowplow Console.

For complete documentation from Amazon go [here](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts.html).

### Set up Role and IAM permissions with **CloudFormation**

We also provide a CloudFormation template that will create a role named SnowplowAdmin with the full permission set [here](https://snowplow-hosted-assets.s3-eu-west-1.amazonaws.com/common/iam/SnowplowAdminRole_CF.yml). The IAM quota will also need updating for this template to apply successfully.

1. Access the CloudFormation service within the sub-account
2. Go to `Stacks` select `Create stack` > `With new resources (standard)`
3. Select `Template is ready` within the Prepare template block
4. Specify an Amazon S3 URL with the full path to the SnowplowAdmin CloudFormation template and proceed
5. Provide the stack with a meaningful name such as SnowplowAdmin stack
6. Now proceed through the remainder of the prompts and choose `Create stack`

For complete documentation from Amazon go [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html).

### Set up the Snowplow deployment role

The last step is to set up the Snowplow deployment role. This is a role assumed by the machine user to make changes with Terraform. Contact Snowplow Support via the Snowplow Console or through your Snowplow account team to request the full list of permissions attached to this role.

1. In the AWS sub-account, open IAM from the console
2. Navigate to `Access management` > `Roles` and click `Create role`
3. Under trusted entity type, select `AWS account`, then choose `Another AWS account`

- Account ID: `793733611312`
- Leave `Require MFA` unchecked — Snowplow uses Okta for MFA, which handles authentication before role assumption

4. If a Permission Boundary was set on the admin role, add it in the bottom section of the permissions page.
5. Name the role `SnowplowDeployment` — this exact name is required. Optionally add a description: "Allows the Snowplow team to programmatically deploy to this account". Create the role.
6. Open the role you just created, go to the **Permissions** tab and click **Add permissions** > **Create inline policy**. Switch to the JSON editor, paste the following policy, and save it:

```json
{

  "Version": "2012-10-17",

  "Statement": [

      {

          "Sid": "IAMBootstrap",

          "Effect": "Allow",

          "Resource": "*",

          "Action": [

              "iam:AddRoleToInstanceProfile",

              "iam:AddUserToGroup",

              "iam:AttachGroupPolicy",

              "iam:AttachRolePolicy",

              "iam:CreateGroup",

              "iam:CreateInstanceProfile",

              "iam:CreatePolicy",

              "iam:CreatePolicyVersion",

              "iam:CreateRole",

              "iam:CreateServiceLinkedRole",

              "iam:CreateUser",

              "iam:GetGroup",

              "iam:GetInstanceProfile",

              "iam:GetPolicy",

              "iam:GetPolicyVersion",

              "iam:GetRole",

              "iam:GetUser",

              "iam:ListAttachedGroupPolicies",

              "iam:ListAttachedRolePolicies",

              "iam:ListGroupsForUser",

              "iam:ListInstanceProfilesForRole",

              "iam:ListPolicyVersions",

              "iam:ListRolePolicies",

              "iam:ListRoleTags",

              "iam:PassRole",

              "iam:TagPolicy",

              "iam:TagRole",

              "iam:TagUser",

              "iam:UpdateAssumeRolePolicy"

          ]

      }

  ]

}
```

7. Go to the **Trust relationships** tab and click **Edit trust policy**. Replace the generated JSON with the following and click **Update policy**:

```json
{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "AWS": "arn:aws:iam::793733611312:root"

      },

      "Action": "sts:AssumeRole",

      "Condition": {

        "StringLike": {

          "aws:PrincipalArn": [

            "arn:aws:iam::793733611312:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_sre-*",

            "arn:aws:iam::793733611312:user/nomad-handler"

          ]

        }

      }

    }

  ]

}
```

7. Copy the Snowplow deployment role ARN. You will need to share this role with Snowplow as part of filling out the setup form in Snowplow Console.

### Provide a CIDR range for VPC peering or using a custom VPC (optional)

For VPC peering, you will need to select a peering-compatible /21 or /22 CIDR range (2048 or 1024 addresses).

If you require Snowplow to be deployed into a custom VPC, you will need to specify a /18 CIDR range (16,384 addresses) for that VPC so that we can create /20 and /23 subnets. This ensures that we have an ample room for scaling, as well as deploying additional services.

Note: VPC peering and using a custom VPC are additional bolt-ons.

### Determine if Snowplow requires a specific IAM Permission Boundary (optional)

If you require Snowplow to be deployed using a specific IAM Permission Boundary, this should be provided at the same time.

### Final checklist

If you are sending a request to Snowplow to set up your account, please ensure you provide the following information:

1. SnowplowAdmin role ARN
2. SnowplowDeployment role ARN
3. AWS region to deploy into
4. VPC CIDR requirements for VPC peering or a custom VPC (optional)
5. The IAM Permission Boundary ARN (optional)

---

# Private Managed Cloud on Azure
> Set up Snowplow CDI Private Managed Cloud on Azure with subscription configuration, service principal permissions, and custom roles.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/setup-guide-azure/

To set up Snowplow, log in to Snowplow [Console](https://console.snowplowanalytics.com) where you'll be able to follow a step-by-step guide to getting started, including the steps below.

## Setting up your Azure account

The following steps explain how to set up your cloud environment ready for your Snowplow pipeline to be installed.

### Create a new subscription

We require a new subscription which does not contain anything else that you have running in Azure. If you don't have an Azure account, then you will need to create a new Azure account and tenant, along with a subscription for Snowplow to use. For the latest documentation on setting up a new account and tenant please refer to the [Azure documentation](https://azure.microsoft.com/).

### Enable billing for the account

Enable billing in the tenant by creating a subscription. Otherwise, the pipeline will fail to deploy. For details on enabling billing for your tenant, please refer to the Azure documentation for [subscriptions](https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription).

### Set up access and required permissions

Snowplow deploys into your tenant using a verified [application service principal](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals?tabs=browser#service-principal-object) (Enterprise application). We require a custom role to be assigned to the application service principal. This will allow us to create custom pipeline roles needed for deploying and managing different components of your infrastructure.

#### Consent to Snowplow Private Managed Cloud Deployment application

You will need to grant our verified application service principal the access into your Azure tenant. Once that’s done, you should see the application service principal under _Microsoft Entra ID_ → _Enterprise Applications_.

1. Grab your Azure tenant ID by navigating to Microsoft Entra ID
2. Complete the consent URL where `<TenantID>` is your tenant ID. The `client_id` set is for “Snowplow BDP Enterprise Deployment” application service principal. Go to this URL in a browser window
   ```text
   https://login.microsoftonline.com/<TenantID>/oauth2/authorize?client_id=0581feb4-b614-42c7-b8e7-b4e7fba9153a&response_type=code
   ```
3. A consent window will appear detailing that an Enterprise application is being set up in your tenant. It needs to be accepted by your Azure tenant admin for the organization (there is a tick box that must be ticked). After accepting, Microsoft redirects you to a page unrelated to the Azure Portal, so close this window
4. Verify the trust has been established by viewing “Snowplow Private Managed Cloud Deployment” application in the Enterprise Applications section of Entra ID

#### Create and assign role to application service principal

Create a custom role and assign it the “Snowplow BDP Enterprise Deployment” application service principal under your subscription. This grants the permission to create distinct roles for deploying and managing infrastructure resources that make up your pipeline.

1. Navigate to your newly created subscription
2. Click into “Access Control (IAM)”
3. Click “Add custom role”
4. Create the following role, which allows Snowplow to create various roles that are specific to components that make up your pipeline. The `customer_subscription_id` should be the subscription ID that Snowplow will deploy into. The role should be single scoped to the subscription. It is recommended to have a custom role per environment to give clear separation between environments.
   ```json
   {

       "properties": {

           "roleName": "Snowplow-Deployment-Role-Creator-Role",

           "description": "Custom policy to allow creation of individual Azure stack related roles",

           "assignableScopes": [

               "/subscriptions/<customer_subscription_id>"

           ],

           "permissions": [

               {

                   "actions": [

                       "Microsoft.Authorization/roleAssignments/write",

                       "Microsoft.Authorization/roleAssignments/read",

                       "Microsoft.Authorization/roleAssignments/delete",

                       "Microsoft.Authorization/roleDefinitions/write",

                       "Microsoft.Authorization/roleDefinitions/read",

                       "Microsoft.Authorization/roleDefinitions/delete",

                       "Microsoft.Resources/subscriptions/resourcegroups/read"

                   ],

                   "notActions": [],

                   "dataActions": [],

                   "notDataActions": []

               }

           ]

       }

   }
   ```
5. Within “Access Control (IAM)”, click “Add role assignment”
6. Assign the `Snowplow-Deployment-Role-Creator-Role` to service principal “Snowplow BDP Enterprise Deployment”. The role can be found under the “Privileged administrator roles” tab. The Conditions tab should be selected and you must select the third option. This is required to enable the [Lighthouse Offer to be created as detailed in Microsoft documentation](https://learn.microsoft.com/en-us/azure/lighthouse/how-to/deploy-policy-remediation#create-a-user-who-can-assign-roles-to-a-managed-identity-in-the-customer-tenant:~:text=To%20allow%20a,Administrator%20or%20Owner) ![Azure role assignment Conditions tab with the third option selected, which allows the Snowplow BDP Enterprise Deployment service principal to assign roles to a managed identity as required for the Lighthouse Offer](/assets/images/azure_role_assignment_conditions-d9c19eb3283061a7a93b07595eb33f38.png)

### Determine if Snowplow requires a specific VPC (optional)

If you require Snowplow to be deployed into a specific VPC CIDR range, this should be provided at the same time as credentials. We need a /16 provided for the VPC so that we can create /20 subnets (note: [VPC peering and using a custom VPC is an additional bolt-on](https://snowplow.io/snowplow-behavioral-data-platform-product-description/#h-vpc-peering-aws-gcp))

### Check subscription resource availability

You must check that the following infrastructure is available in the subscription before deployment starts. If the required instances are unavailable, it will result in delays to getting started.

Snowplow requires the below instances and databases. We have provided the Azure CLI commands that can be used to check their availability. Please note that they are required in all three availability zones in the region for redundancy. You may need to raise a quota ticket with Azure or contact your Azure Technical Account Manager to ensure the instance types are fully available.

Instances D4s\_v5 (preferred), or D4s\_v4 / D4\_v5 / D4\_v4 series:

```bash
az vm list-skus --location <REGION> --size Standard_D4s_v5 --all --output table

az vm list-skus --location <REGION> --size Standard_D4s_v4 --all --output table

az vm list-skus --location <REGION> --size Standard_D4_v5 --all --output table

az vm list-skus --location <REGION> --size Standard_D4_v4 --all --output table
```

Database B\_Standard\_B2s:

```bash
az vm list-skus --location <REGION> --size Standard_B2s --all --output table
```

Please note that the above is for a minimal deployment. If you will be sending over one billion events per month through the pipeline, please speak to us for advice on what instance sizes are recommended.

### Final checklist

If you are sending a request to our team to set up deployment into your Azure account, please ensure you provide the following information:

1. The tenant ID
2. The subscription ID
3. Azure region to deploy into
4. The ID of the `Snowplow-Deployment-Role-Creator-Role`
5. The specific VPC CIDR range that the pipeline should be deployed in (optional).

---

# Private Managed Cloud on GCP
> Set up Snowplow CDI Private Managed Cloud on GCP with project configuration, IAM roles, and service account permissions.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/setup-guide-gcp/

To set up Snowplow, log in to Snowplow [Console](https://console.snowplowanalytics.com) where you'll be able to follow a step-by-step guide to getting started, including the steps below.

## Setting up your Google project

The following steps explain how to set up your cloud environment ready for your Snowplow pipeline to be installed.

### Create a new project

Please create a new project segregated from anything else you have running in GCP. For the latest documentation on setting up a new project please refer to Google Cloud's documentation [here](https://cloud.google.com/resource-manager/docs/creating-managing-projects).

### Set up required permissions

Please add **<techops-cloud-admin@snowplowanalytics.com>** to your project. This account will be used to install and maintain your pipeline in GCP. The following roles are required:

- [`editor`](https://cloud.google.com/iam/docs/roles-overview#basic-definitions)
- [`Roles/resourcemanager.projectIamAdmin`](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
- [`Roles/errorreporting.admin`](https://cloud.google.com/iam/docs/understanding-roles#errorreporting.admin)
- [`Roles/logging.admin`](https://cloud.google.com/iam/docs/understanding-roles#logging.admin)
- [`Roles/monitoring.admin`](https://cloud.google.com/iam/docs/understanding-roles#monitoring.admin)
- [`Roles/iam.roleAdmin`](https://cloud.google.com/iam/docs/understanding-roles#iam.roleAdmin)
- [`Roles/iam.serviceAccountAdmin`](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountAdmin)
- [`Roles/container.admin`](https://cloud.google.com/iam/docs/roles-permissions/container#container.admin)
- [`Roles/run.admin`](https://cloud.google.com/run/docs/reference/iam/roles#run.admin)
- [`Roles/datastore.owner`](https://docs.cloud.google.com/iam/docs/roles-permissions/firestore#datastore.owner)

The following roles are also required if using [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) within GCP:

- [`Roles/secretmanager.secretAccessor`](https://cloud.google.com/iam/docs/understanding-roles#secretmanager.secretAccessor)
- [`Roles/secretmanager.secretVersionAdder`](https://cloud.google.com/iam/docs/understanding-roles#secretmanager.secretVersionAdder)

### Enable billing for the project

Please enable billing on the project as failing to do so will prevent the pipeline from being installed. For details on enabling billing for your project, please refer to the GCP documentation for [APIs and billing](https://support.google.com/googleapi/answer/6158867).

### Ensure that the following constraints are turned off

The following is a list of Organizational Policy constraints that we require to be turned off, otherwise we are unable to deploy the pipeline. For some, more specific configuration would enable the constraint to stay and the pipeline to deploy. This list will be updated as we discover more constraints that are not compatible with the pipeline deployment.

- `constraints/compute.requireShieldedVm`
- `constraints/gcp.resourceLocations`
- `constraints/iam.disableServiceAccountCreation`
- `constraints/iam.disableServiceAccountKeyCreation`
- `constraints/sql.restrictPublicIp`
- `constraints/sql.restrictAuthorizedNetworks`
- `constraints/sql.restrictNoncompliantResourceCreation`

---

# Querying your first events
> Inspecting the events you tracked
> Source: https://docs.snowplow.io/docs/get-started/querying/

Once you've tracked some events, you'll want to look at them in your data warehouse, database, or lake. The exact steps will depend on your choice of storage and your Snowplow platform.

For **Snowplow CDI** Private Managed Cloud or Cloud customers, you can find your connection details in [Snowplow Console](https://console.snowplowanalytics.com/destinations/warehouses-lakes), under the destination you've selected.

Follow [our querying guide](/docs/destinations/warehouses-lakes/querying-data/) for advice on querying your data.

## Snowplow Self-Hosted

If you don't have access to Snowplow Console, follow these instructions to connect to your Snowplow data:

**Postgres:**

Your database will be named according to the `postgres_db_name` Terraform variable. It will contain two schemas:

- `atomic` — the validated events
- `atomic_bad` — the [failed events](/docs/fundamentals/failed-events/)

You can connect to the database using the credentials you provided for the loader in the Terraform variables (`postgres_db_username` and `postgres_db_password`), along with the `postgres_db_address` and `postgres_db_port` Terraform outputs.

> **Tip:** If you need to reset your username or password, you can follow [these steps](https://aws.amazon.com/premiumsupport/knowledge-center/reset-master-user-password-rds/).

See [the AWS RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html) for more details on how to connect.

> **Note:** If you opted for the `secure` option, you will first need to create a tunnel into your VPC to be able to connect to your RDS instance and be able to query the data. A common solution to this issue is to configure a bastion host as described [here](https://repost.aws/knowledge-center/rds-connect-using-bastion-host-linux).

**Redshift:**

The database name and the schema name will be defined by the `redshift_database` and `redshift_schema` variables in Terraform.

There are two different ways to login to the database:

- The first option is to use the credentials you configured for the loader in the Terraform variables (`redshift_loader_user` and `redshift_loader_password`)
- The second option is to grant `SELECT` permissions on the schema to an existing user

To connect, you can use the Redshift UI or something like [`psql`](https://www.postgresql.org/docs/current/app-psql.html).

**BigQuery:**

The database will be called `<prefix>_snowplow_db`, where `<prefix>` is the prefix you picked in your Terraform variables file. It will contain an `atomic` schema with your validated events.

You can access the database via the [BigQuery UI](https://console.cloud.google.com/bigquery).

**Snowflake:**

The database name and the schema name will be defined by the `snowflake_database` and `snowflake_schema` variables in Terraform.

There are two different ways to login to the database:

- The first option is to use the credentials you configured for the loader in the Terraform variables (`snowflake_loader_user` and `snowflake_loader_password`)
- The second option is to grant `SELECT` permissions on the schema to an existing user

To connect, you can use either Snowflake dashboard or [SnowSQL](https://docs.snowflake.com/en/user-guide/snowsql.html).

**Databricks:**

> **Info:** On Azure, you have created an external table in the [last step of the guide](/docs/get-started/self-hosted/quick-start/#configure-the-destination). Use this table and ignore the text below.

The database name and the schema name will be defined by the `databricks_database` and `databricks_schema` variables in Terraform.

There are two different ways to login to the database:

- The first option is to use the credentials you configured for the loader in the Terraform variables (`databricks_loader_user` and `databricks_loader_password`, or alternatively the `databricks_auth_token`)
- The second option is to grant `SELECT` permissions on the schema to an existing user

See the [Databricks tutorial](https://docs.databricks.com/getting-started/quick-start.html) for more details on how to connect. The documentation on [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/queries.html) is also useful.

**Synapse Analytics:**

In Synapse Analytics, you can connect directly to the data residing in ADLS. You will need to know the names of the storage account (set in the `storage_account_name` Terraform variable) and the storage container (it’s a fixed value: `lake-container`).

Follow [the Synapse documentation](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format) and use the `OPENROWSET` function. If you created a data source in the [last step](/docs/get-started/self-hosted/quick-start/#configure-the-destination) of the quick start guide, your queries will be a bit simpler.

> **Tip:** If you created a OneLake shortcut in the [last step](/docs/get-started/self-hosted/quick-start/#configure-the-destination) of the quick start guide, you will be able to explore Snowplow data in Fabric, for example, using Spark SQL.

***

---

# Snowplow Self-Hosted quick start: frequently asked questions
> Common questions about managing, upgrading, and troubleshooting Snowplow Self-Hosted deployments with Terraform.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/faq/

## How do I shut down the pipeline?

If you would like to shut down your pipeline then you can easily do so by running `terraform destroy`.

Note that if you want to delete your S3 bucket or Postgres database, you would need to do that from within the AWS, GCP or Azure console. If you want to maintain these then you can - just be aware that next time you spin up your pipeline you might see some errors.

## How do I upgrade the version of the application that I am using?

We release new versions of our pipeline components very frequently; however the versions used within the terraform modules are updated a few times a year to the most stable and recommended versions of our components.

When a new version of a module is released, follow these instructions to upgrade:

- Update the module version in your terraform
- Run `terraform plan` to check for what changes will be made

## Which enrichments are enabled by default?

The following enrichments are enabled by default within the Enrich module:

- [UA parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)
- [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)
- [Campaign Attribution](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)
- [Event fingerprint](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)
- [Referer parser](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)

Other available enrichments enrichments and the configurations can be found [here](/docs/pipeline/enrichments/available-enrichments/).

Follow the [documentation](/docs/pipeline/enrichments/managing-enrichments/terraform/) if you want to enable or disable certain enrichments.

## Troubleshooting Terraform Errors

The following are some common errors that you might encounter when running `terraform plan` or `terraform apply`.

AWS:

**Error: Invalid provider configuration**

`Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described in the provider documentation.`

**Solution:** Double check that your AWS Access Key ID and AWS Secret Access Key are set up correctly (with no typos) using the `aws configure` command.

**Error:** **"x.x.x.x" is not a valid CIDR block**

`"x.x.x.x" is not a valid CIDR block: invalid CIDR address: x.x.x.x with module.iglu_server.aws_security_group_rule.ingress_tcp_22, on .terraform/modules/iglu_server/main.tf line 143, in resource "aws_security_group_rule" "ingress_tcp_22": 143: cidr_blocks = var.ssh_ip_allowlist`

**Solution**: Add a mask to the IP in your terraform.tvars file, e.g. `x.x.x.x/32`.

**Error creating Application Load Balancer**

`ValidationError: At least two subnets in two different Availability Zones must be specified status code: 400`

**Solution**: Add subnets to cover at least 2 availability zones. See this [AWS guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-public-private-vpc.html) on how to set this up.

After this step, add the two freshly created subnets to your `terraform.tfvars` file like this: `public_subnet_ids = ["subnet-00000000", "subnet-00000001"]` and run `terraform apply` again.

**Error creating DB Subnet Group**

`DBSubnetGroupDoesNotCoverEnoughAZs: DB Subnet Group doesn't meet availability zone coverage requirement. Please add subnets to cover at least 2 availability zones. Current coverage: 1 status code: 400`

**Solution**: Add subnets to cover at least 2 availability zones. See this [AWS guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-public-private-vpc.html) on how to set this up.

After this step, add the two freshly created subnets to your `terraform.tfvars` file like this: `public_subnet_ids = ["subnet-00000000", "subnet-00000001"]` and run `terraform apply` again.

**Error creating DB Instance: InvalidParameterValue**

`The parameter MasterUserPassword is not a valid password. Only printable ASCII characters besides '/', '@', '"', ' ' may be used.`

**Solution**: Modify the `iglu_db_password` in your `terraform.tfvars` file so that it does not contain any forbidden characters like '\_', '\*'. Make sure the password is not longer than 13 characters.

---

# Set up Snowplow Self-Hosted
> Deploy Snowplow Community Edition using Terraform modules on AWS, GCP, or Azure for non-production testing and evaluation.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/

This page is an introduction for the [Self-Hosted](/docs/get-started/#self-hosted) [Quick Start guide](/docs/get-started/self-hosted/quick-start/). Follow the Quick Start to set up a self-hosted deployment.

We have built a set of [terraform](https://www.terraform.io/docs/language/modules/develop/index.html) modules, which automates the setup and deployment of the required infrastructure and applications for an operational Snowplow Community Edition pipeline, with just a handful of input variables required on your side.

After following this guide, you will be able to: 

- Collect granular, well-structured data with our suite of web, mobile and server side [SDKs](/docs/sources/)
- Create your own [custom events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/#custom-entities)
- Easily enable and disable our suite of [out-of-the-box enrichments](/docs/pipeline/enrichments/available-enrichments/)
- Consume your rich data from the data warehouse, database, lake and/or real-time stream

> **Note:** Please read the terms of the [Snowplow Limited Use License Agreement](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)) which apply to Community Edition.
>
> In short, Community Edition is meant for testing and evaluating Snowplow and must not be deployed in production.

## Required time

If you are proficient with Terraform and cloud tooling, 1 hour should be sufficient. Otherwise, expect to spend a few hours.

## Cost

Assuming around 100 events per second, the pipeline will cost around $200 per month on AWS and $240 per month on GCP.

> **Tip:** To reduce the costs, you can tweak the configuration (e.g. use smaller instances), or shut down the pipeline when not in use.

## Scale

Out of the box, the deployed pipeline will handle up to \~100 events per second (\~9 million events per day).

## Getting help

Check out our [Community](https://community.snowplow.io/). If you run into any problems or have any questions, we are here to help.

---

# Quick start guide for Snowplow Self-Hosted
> Deploy Snowplow Community Edition on AWS, GCP, or Azure using Terraform modules with step-by-step instructions.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/quick-start/

This guide will take you through how to spin up a Snowplow Community Edition pipeline using the [Snowplow Terraform modules](https://registry.terraform.io/namespaces/snowplow-devops). _(Not familiar with Terraform? Take a look at [Infrastructure as code with Terraform](https://learn.hashicorp.com/tutorials/terraform/infrastructure-as-code?in=terraform/aws-get-started).)_

## Prerequisites

[Sign up](https://snowplow.io/pricing/) for Snowplow Community Edition and follow the link in the email to get a copy of the repository containing the Terraform code.

> **Note:** Please read the terms of the [Snowplow Limited Use License Agreement](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)) which apply to Community Edition.
>
> In short, Community Edition is meant for testing and evaluating Snowplow and must not be deployed in production.

Install [Terraform 1.0.0](https://www.terraform.io/downloads.html) or higher. Follow the instructions to make sure the `terraform` binary is available on your `PATH`. You can also use [tfenv](https://github.com/tfutils/tfenv) to manage your Terraform installation.

**AWS:**

Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) version 2.

[Configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) the CLI against a role that has the `AdminstratorAccess` policy attached.

> **Warning:** `AdminstratorAccess` allows all actions on all AWS services and shouldn't be used in production

Details on how to configure the AWS Terraform Provider can be found [on the registry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables).

**GCP:**

Install [Google Cloud SDK](https://cloud.google.com/sdk/docs/install).

Make sure the following APIs are active in your GCP account (this list might not be exhaustive and is subject to change as GCP APIs evolve):

- [Compute Engine API](https://console.cloud.google.com/apis/api/compute.googleapis.com)
- [Cloud Resource Manager API](https://console.cloud.google.com/apis/api/cloudresourcemanager.googleapis.com)
- [Identity and Access Management (IAM) API](https://console.cloud.google.com/apis/api/iam.googleapis.com)
- [Cloud Pub/Sub API](https://console.cloud.google.com/apis/api/pubsub.googleapis.com)
- [Cloud SQL Admin API](https://console.cloud.google.com/apis/api/sqladmin.googleapis.com)

Configure a Google Cloud service account. See [details on using the service account with the Cloud SDK](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable). You will need to:

- Navigate to your service account on Google Cloud Console
- Create a new JSON Key and store it locally
- Create the environment variable by running `export GOOGLE_APPLICATION_CREDENTIALS="KEY PATH"` in your terminal

**Azure:**

Install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli).

If your organisation has an existing Azure account, make sure your user has been granted the following roles on a valid Azure Subscription:

- [Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor)
- [User Access Administrator](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#user-access-administrator)
- [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor)

> **Warning:** `User Access Administrator` allows the user to modify, create and delete permissions across Azure resources, and shouldn’t be used in production. Instead, you can use a custom role with the following permissions:
>
> - `Microsoft.Authorization/roleAssignments/write` to deploy the stacks below
> - `Microsoft.Authorization/roleAssignments/delete` to destroy them

If you don’t have an Azure account yet, you can get started with a new [pay-as-you-go account](https://azure.microsoft.com/free/).

Details on how to configure the Azure Terraform Provider can be found [on the registry](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/azure_cli).

***

## Storage options

The sections below will guide you through setting up your destination to receive Snowplow data, but for now here is an overview.

| Warehouse         | AWS | GCP | Azure |
| ----------------- | --- | --- | ----- |
| Snowflake         | ✅   | ❌   | ❌     |
| Databricks        | ✅   | ❌   | ✅     |
| Redshift          | ✅   | —   | —     |
| BigQuery          | —   | ✅   | —     |
| Synapse Analytics | —   | —   | ✅     |

## Real-time streaming options

As part of the deployment, your data will be available in real-time streams corresponding to the cloud provider you have chosen. You can consume data directly from these streams, either in addition to or instead of the data warehouse.

| Stream    | AWS | GCP | Azure |
| --------- | --- | --- | ----- |
| Kinesis   | ✅   | ❌   | ❌     |
| Pub/Sub   | ❌   | ✅   | ❌     |
| EventHubs | ❌   | ❌   | ✅     |

For an out-of-the-box solution to accessing this data in real-time streams, you can [check out our Snowbridge tool](/docs/api-reference/snowbridge/). Alternatively, if you want to develop a custom consumer, you can [leverage our Analytics SDKs](/docs/api-reference/analytics-sdk/) to parse the event formats more easily.

> **Note:** EventHubs topics are deployed in a Kafka-compatible model, so you can consume from them using standard Kafka connector libraries.

**AWS:**

There are three main storage options for you to select: Redshift, Snowflake and Databricks. Additionally, there is an S3 option, which is primarily used to archive enriched events and to store [failed events](/docs/fundamentals/failed-events/).

We recommend to only load data into a single destination, but nothing prevents you from loading into multiple destinations with the same pipeline (e.g. for testing purposes).

**GCP:**

Only one storage option is provided: BigQuery.

**Azure:**

Only a data lake (ADLS) storage option is provided. This enables querying data from Databricks and Synapse Analytics.

***

## Set up a VPC to deploy into

**AWS:**

AWS provides a [default VPC](https://docs.aws.amazon.com/vpc/latest/userguide/default-vpc.html#view-default-vpc) in every region for your sub-account. Take a note of the identifiers of this VPC and the associated subnets for later parts of the deployment.

**GCP:**

GCP provides a [default VPC](https://cloud.google.com/vpc/docs/vpc#default-network) for your project. In the steps below, it is sufficient to set `network = default` and leave subnetworks empty, and Terraform will discover the correct network to deploy into.

**Azure:**

Azure does not provide a default VPC or resource group, so we have added a helper module to create a working network we can deploy into.

To use our out-of-the-box network, you will need to navigate to the `terraform/azure/base` directory in the code repository and update the input variables in `terraform.tfvars`.

Once that’s done, you can use Terraform to create your base network.

```bash
terraform init

terraform plan

terraform apply
```

After the deployment completes, you should get an output like this:

```text
...

vnet_subnets_name_id = {

  "collector-agw1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/collector-agw1"

  "iglu-agw1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/iglu-agw1"

  "iglu-vmss1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/iglu-vmss1"

  "iglu1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/iglu1"

  "pipeline1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/pipeline1"

}
```

These are the subnet identifiers, e.g. `"/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/pipeline1"` is the identifier of the `pipeline1` subnet. Take note of these four identifiers, as you will need them in the following steps.

***

## Set up Iglu Server

The first step is to set up the [Iglu Server](/docs/api-reference/iglu/) stack required by the rest of your pipeline.

This will allow you to create and evolve your own [custom events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/#custom-entities). Iglu Server stores the [schemas](/docs/fundamentals/schemas/) for your events and entities and fetches them as your events are processed by the pipeline.

### Step 1: Update the `iglu_server` input variables

Once you have cloned the code repository, you will need to navigate to the `iglu_server` directory to update the input variables in `terraform.tfvars`.

**AWS:**

```bash
cd terraform/aws/iglu_server/default

nano terraform.tfvars # or other text editor of your choosing
```

**GCP:**

```bash
cd terraform/gcp/iglu_server/default

nano terraform.tfvars # or other text editor of your choosing
```

**Azure:**

```bash
cd terraform/azure/iglu_server

nano terraform.tfvars # or other text editor of your choosing
```

If you [used our `base` module](#set-up-a-vpc-to-deploy-into), you will need to set these variables as follows:

- `resource_group_name`: use the same value as you supplied in `base`
- `subnet_id_database`: use the identifier of the `iglu1` subnet from `base`
- `subnet_id_servers`: use the identifier of the `iglu-vmss1` subnet from `base`

***

To update your input variables, you’ll need to know a few things:

- Your IP Address. [Help](https://duckduckgo.com/?q=ip+address\&t=ffab\&ia=answer).
- A UUIDv4 to be used as the Iglu Server’s API Key. [Help](https://duckduckgo.com/?t=ffab\&q=uuid\&ia=answer).
- How to generate an SSH Key.

> **Tip:** On most systems, you can generate an SSH Key with: `ssh-keygen -t rsa -b 4096`. This will output where you public key is stored, for example: `~/.ssh/id_rsa.pub`. You can get the value with `cat ~/.ssh/id_rsa.pub`.

**Telemetry notice**

By default, Snowplow collects telemetry data for each of the Quick Start Terraform modules. Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `user_provided_id` variable.

If you wish to disable telemetry, you can do so by setting `telemetry_enabled` to `false`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

### Step 2: Run the `iglu_server` Terraform script

You can now use Terraform to create your Iglu Server stack.

**AWS:**

You will be asked to select a region, you can find more information about [available AWS regions here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).

```bash
terraform init

terraform plan

terraform apply
```

The deployment will take roughly 15 minutes.

**GCP:**

```bash
terraform init

terraform plan

terraform apply
```

**Azure:**

```bash
terraform init

terraform plan

terraform apply
```

***

Once the deployment is done, it will output `iglu_server_dns_name`. Make a note of this, you’ll need it when setting up your pipeline. If you have attached a custom SSL certificate and set up your own DNS records, then you don’t need this value.

## Prepare the destination

Depending on the destination(s) you’ve choosen, you might need to perform a few extra steps to prepare for loading data there.

> **Tip:** Feel free to go ahead with these while your Iglu Server stack is deploying.

**Redshift:**

Assuming you already have an active Redshift cluster, execute the following SQL (replace the `${...}` variables with your desired values). You will need the permissions to create databases, users and schemas in the cluster.

```sql
-- 1. (Optional) Create a new database - you can also use an existing one if you prefer

CREATE DATABASE ${redshift_database};

-- Log back into Redshift with the new database:

-- psql --host <host> --port <port> --username <admin> --dbname ${redshift_database}



-- 2. Create a schema within the database

CREATE SCHEMA IF NOT EXISTS ${redshift_schema};



-- 3. Create the loader user

CREATE USER ${redshift_loader_user} WITH PASSWORD '${redshift_password}';



-- 4. Ensure the schema is owned by the loader user

ALTER SCHEMA ${redshift_schema} OWNER TO ${redshift_loader_user};
```

> **Note:** You will need to ensure that the loader can access the Redshift cluster over whatever port is configured for the cluster (usually, `5439`).

**BigQuery:**

No extra steps needed.

**Snowflake:**

You will need to generate a key pair following the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth#generate-the-private-key). Make sure to enter an empty passphrase, as the terraform module below does not support keys with passphrases (for simplicity).

Execute the following SQL (replace the `${...}` variables with your desired values). You will need access to both `SYSADMIN` and `SECURITYADMIN` level roles to action this:

```sql
-- 1. (Optional) Create a new database - you can also use an existing one if you prefer

CREATE DATABASE IF NOT EXISTS ${snowflake_database};



-- 2. Create a schema within the database

CREATE SCHEMA IF NOT EXISTS ${snowflake_database}.${snowflake_schema};



-- 3. Create a role that will be used for loading data

CREATE ROLE IF NOT EXISTS ${snowflake_loader_role};

GRANT USAGE ON DATABASE ${snowflake_database} TO ROLE ${snowflake_loader_role};

GRANT ALL ON SCHEMA ${snowflake_database}.${snowflake_schema} TO ROLE ${snowflake_loader_role};



-- 4. Create a user that can be used for loading data

CREATE USER IF NOT EXISTS ${snowflake_loader_user}

  RSA_PUBLIC_KEY='MIIBIj...'

  TYPE = SERVICE

  DEFAULT_ROLE = ${snowflake_loader_role}

  EMAIL = 'loader@acme.com';

GRANT ROLE ${snowflake_loader_role} TO USER ${snowflake_loader_user};



-- 5. (Optional) Grant this role to SYSADMIN to make debugging easier from admin users

GRANT ROLE ${snowflake_loader_role} TO ROLE SYSADMIN;
```

**Databricks:**

> **Info:** On Azure, we currently support loading data into Databricks via a data lake. You can still follow Step 1 below to create the cluster, however you should skip the rest of these steps. Instead, proceed with [deploying the pipeline](#set-up-the-pipeline) — we will return to configuring Databricks [at the end of this guide](#configure-the-destination).

#### Step 1: Create a cluster

> **Note:** The cluster spec described below should be sufficient for a monthly event volume of up to 10 million events. If your event volume is greater, then you may need to increase the size of the cluster.

Create a new cluster, following the [Databricks documentation](https://docs.databricks.com/clusters/create.html), with the following settings:

- the runtime version must be 13.0 or greater (but not 13.1 or 13.2)
- single node cluster
- "smallest" size node type
- auto-terminate after 30 minutes.

**Advanced cluster configuration (optional)**

You might want to configure cluster-level permissions, by following [the Databricks instructions on cluster access control](https://docs.databricks.com/security/access-control/cluster-acl.html). Snowplow's RDB Loader must be able to restart the cluster if it is terminated.

If you use AWS Glue Data Catalog as your metastore, [follow these Databricks instructions](https://docs.databricks.com/data/metastores/aws-glue-metastore.html) for the relevant spark configurations. You will need to set `spark.databricks.hive.metastore.glueCatalog.enabled true` and `spark.hadoop.hive.metastore.glue.catalogid <aws-account-id-for-glue-catalog>` in the spark configuration.

You can configure your cluster with [an instance profile](https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html) if it needs extra permissions to access resources. For example, if the S3 bucket holding the delta lake is in a different AWS account.

#### Step 2: Note the JDBC connection details

1. In the Databricks UI, click on "Compute" in the sidebar
2. Click on the _RDB Loader cluster_ and navigate to "Advanced options"
3. Click on the "JDBC/ODBC" tab
4. Note down the JDBC connection URL, specifically the `host`, the `port` and the `http_path`

#### Step 3: Create an access token for the loader

> **Note:** The access token must not have a specified lifetime. Otherwise, the loader will stop working when the token expires.

1. Navigate to the user settings in your Databricks workspace

   - For Databricks hosted on AWS, the "Settings" link is in the lower left corner in the side panel
   - For Databricks hosted on Azure, "User Settings" is an option in the drop-down menu in the top right corner.

2. Go to the "Access Tokens" tab

3. Click the "Generate New Token" button

4. Optionally enter a description (comment). Leave the expiration period empty

5. Click the "Generate" button

6. Copy the generated token and store it in a secure location

#### Step 4: Create the catalog and the schema

Execute the following SQL (replace the `${...}` variables with your desired values). The default catalog is called `hive_metastore` and is what you should use in the loader unless you specify your own.

```sql
-- USE CATALOG ${catalog_name}; -- Uncomment if your want to use a custom Unity catalog and replace with your own value.



CREATE SCHEMA IF NOT EXISTS ${schema_name}

-- LOCATION s3://<custom_location>/ -- Uncomment if you want tables created by Snowplow to be located in a non-default bucket or directory.

;
```

**Advanced security configuration (optional)**

The security principal used by the loader needs a `Databricks SQL access` permission, which can be enabled in the _Admin Console_.

Databricks does not have table access enabled by default. Enable it with an initialization script:

```scala
dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""

|#!/bin/bash

|

|cat << 'EOF' > /databricks/driver/conf/00-custom-table-access.conf

|[driver] {

|  "spark.databricks.acl.sqlOnly" = "true"

|}

|EOF

""".stripMargin, true)
```

After adding the script, you need to restart the cluster. Verify that changes took effect by evaluating `spark.conf.get("spark.databricks.acl.sqlOnly")`, which should return `true`.

Follow the rest of the quick start guide so that the loader creates the required tables. Afterwards, reconfigure the permissions:

```sql
-- Clean initial permissions from tables

REVOKE ALL PRIVILEGES ON TABLE <catalog>.<schema>.events FROM `<principal>`;

REVOKE ALL PRIVILEGES ON TABLE <catalog>.<schema>.manifest FROM `<principal>`;

REVOKE ALL PRIVILEGES ON TABLE <catalog>.<schema>.rdb_folder_monitoring FROM `<principal>`;



-- Clean initial permissions from schema

REVOKE ALL PRIVILEGES ON SCHEMA <catalog>.<schema> FROM `<principal>`;



-- Loader will run CREATE TABLE IF NOT EXISTS statements, so USAGE and CREATE and both required.

GRANT USAGE, CREATE ON SCHEMA <catalog>.<schema> TO `<principal>`;



-- COPY TO statement requires ANY FILE and MODIFY for the receiving table

GRANT SELECT ON ANY FILE TO `<principal>`;

GRANT MODIFY  ON TABLE  <catalog>.<schema>.events TO `<principal>`;



-- These tables are used to store internal loader state

GRANT MODIFY, SELECT ON TABLE  <catalog>.<schema>.manifest TO `<principal>`;

GRANT MODIFY, SELECT ON TABLE  <catalog>.<schema>.rdb_folder_monitoring TO `<principal>`;
```

**Synapse Analytics:**

No extra steps needed. Proceed with [deploying the pipeline](#set-up-the-pipeline) — we will return to configuring Synapse [at the end of this guide](#configure-the-destination).

***

## Set up the pipeline

In this section, you will update the input variables for the Terraform module, and then run the Terraform script to set up your pipeline. At the end you will have a working Snowplow pipeline ready to receive web, mobile or server-side data.

### Step 1: Update the `pipeline` input variables

Navigate to the `pipeline` directory in the code repository and update the input variables in `terraform.tfvars`.

**AWS:**

```bash
cd terraform/aws/pipeline/default

nano terraform.tfvars # or other text editor of your choosing
```

**GCP:**

```bash
cd terraform/gcp/pipeline/default

nano terraform.tfvars # or other text editor of your choosing
```

**Azure:**

```bash
cd terraform/azure/pipeline

nano terraform.tfvars # or other text editor of your choosing
```

If you [used our `base` module](#set-up-a-vpc-to-deploy-into), you will need to set these variables as follows:

- `resource_group_name`: use the same value as you supplied in `base`
- `subnet_id_lb`: use the identifier of the `collector-agw1` subnet from `base`
- `subnet_id_servers`: use the identifier of the `pipeline1` subnet from `base`

**Confluent Cloud**

If you already use Confluent Cloud, you can opt to create the necessary message topics there, instead of relying on Azure Event Hubs. This way, you will also benefit from features like [Stream Lineage](https://docs.confluent.io/cloud/current/stream-governance/stream-lineage.html).

To do this, you will need to:

- Set the `stream_type` variable to `confluent_cloud`
- Create 3 or 4 topics manually in your Confluent cluster and add their names in the respective variables (`confluent_cloud_..._topic_name`)
- Create an API key and fill the relevant fields (`confluent_cloud_api_key`, `confluent_cloud_api_secret`)
- Add a bootstrap server in `confluent_cloud_bootstrap_server`

> **Tip:** If you need to stay within the free tier for your Confluent cluster, make sure to select no more than 2 partitions for each topic.

***

To update your input variables, you’ll need to know a few things:

- Your IP Address. [Help](https://duckduckgo.com/?q=ip+address\&t=ffab\&ia=answer).
- Your Iglu Server’s domain name from the [previous step](#step-2-run-the-iglu_server-terraform-script)
- Your Iglu Server’s API Key from the [previous step](#step-1-update-the-iglu_server-input-variables)
- How to generate an SSH Key.

> **Tip:** On most systems, you can generate an SSH Key with: `ssh-keygen -t rsa -b 4096`. This will output where you public key is stored, for example: `~/.ssh/id_rsa.pub`. You can get the value with `cat ~/.ssh/id_rsa.pub`.

#### Destination-specific variables

**AWS:**

As mentioned [above](#storage-options), there are several options for the pipeline’s destination database. For each destination you’d like to configure, set the `<destination>_enabled` variable (e.g. `redshift_enabled`) to `true` and fill all the relevant configuration options (starting with `<destination>_`).

When in doubt, refer back to the [destination setup](#prepare-the-destination) section where you have picked values for many of the variables.

**Snowflake**

If you are using Snowflake, you will need to provide a private key you’ve generated during [destination setup](#prepare-the-destination).

Here’s how to do it:

```text
snowflake_streaming_loader_private_key = <<EOT

-----BEGIN PRIVATE KEY-----

MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCd2dEYSUp3hdyK

5hWwpkNGG56hLFWDK47oMf/Niu+Yh+8Wm4p9TlPje+UuKOnK5N4nAbM4hlhKyEJv

...

99Xil8uas3v7o2OSe7FfLA==

-----END PRIVATE KEY-----

EOT
```

> **Note:** For all active destinations, change any `_password` setting to a value that _only you_ know.

**GCP:**

Set `bigquery_db_enabled` to `true` to enable loading to BigQuery.

**Azure:**

Set `lake_enabled` to `true` to enable loading to a data lake.

***

### Step 2: Run the `pipeline` Terraform script

**AWS:**

You will be asked to select a region, you can find more information about [available AWS regions here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).

```bash
terraform init

terraform plan

terraform apply
```

This will output your `collector_dns_name`.

**GCP:**

```bash
terraform init

terraform plan

terraform apply
```

This will output your `collector_ip_address`, `bigquery_db_dataset_id` and `bq_loader_bad_rows_topic_name`.

**Azure:**

```bash
terraform init

terraform plan

terraform apply
```

This will output your `collector_lb_ip_address` and `collector_lb_fqdn`.

***

Make a note of the outputs: you'll need them when sending events and (in some cases) connecting to your data.

> **Tip:** Depending on your cloud and chosen destination, some of these outputs might be empty — you can ignore those.

If you have attached a custom SSL certificate and set up your own DNS records, then you don't need `collector_dns_name`, as you will use your own DNS record to send events from the Snowplow trackers.

> **Tip:** For solutions to some common Terraform errors that you might encounter when running `terraform plan` or `terraform apply`, see the [FAQs section](/docs/get-started/self-hosted/faq/#troubleshooting-terraform-errors).

## Configure the destination

**Redshift:**

No extra steps needed.

**BigQuery:**

No extra steps needed.

**Snowflake:**

No extra steps needed.

**Databricks:**

> **Info:** On Azure, we currently support loading data into Databricks via a data lake. To complete the setup, you will need to configure Databricks to access your data on ADLS.
>
> First, follow the [Databricks documentation](https://docs.databricks.com/sap/en/external-locations-azure) to set up authentication using either Azure service principal, shared access signature tokens or account keys. _(The latter mechanism is not recommended, but is arguably the easiest for testing purposes.)_
>
> You will need to know a couple of things:
>
> - Storage account name — this is the value of the `storage_account_name` variable in the pipeline `terraform.tvars` file
> - Storage container name — `lake-container`
>
> Once authentication is set up, you can create an external table using Spark SQL (replace `<storage-account-name>` with the corredponding value):
>
> ```sql
> CREATE TABLE events
>
> LOCATION 'abfss://lake-container@<storage-account-name>.dfs.core.windows.net/events/';
> ```

**Synapse Analytics:**

Your data is loaded into ADLS. To access it, follow [the Synapse documentation](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format) and use the `OPENROWSET` function.

You will need to know a couple of things:

- Storage account name — this is the value of the `storage_account_name` variable in the pipeline `terraform.tvars` file
- Storage container name — `lake-container`

**Example query**

```sql
SELECT TOP 10 *

FROM OPENROWSET(

    BULK 'https://<storage-account-name>.blob.core.windows.net/lake-container/events/',

    FORMAT = 'delta'

) AS events;
```

We recommend [creating a data source](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format#data-source-usage), which simplifies future queries (note that unlike the previous URL, this one does not end with `/events/`):

```sql
CREATE EXTERNAL DATA SOURCE SnowplowData

WITH (LOCATION = 'https://<storage-account-name>.blob.core.windows.net/lake-container/');
```

**Example query with data source**

```sql
SELECT TOP 10 *

FROM OPENROWSET(

    BULK 'events',

    DATA_SOURCE = 'SnowplowData',

    FORMAT = 'delta'

) AS events;
```

> **Tip:** You can also consume your ADLS data via Fabric and OneLake:
>
> - First, [create a Lakehouse](https://learn.microsoft.com/en-us/fabric/onelake/create-lakehouse-onelake#create-a-lakehouse) or use an existing one.
> - Next, [create a OneLake shortcut](https://learn.microsoft.com/en-us/fabric/onelake/create-adls-shortcut) to your storage account. In the URL field, specify `https://<storage-account-name>.blob.core.windows.net/lake-container/events/`.
> - You can now [use Spark notebooks](https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-notebook-explore) to explore your Snowplow data.
>
> Do note that currently not all Fabric services support nested fields present in the Snowplow data.

***

## Configure HTTPS (optional)

Now that you have a working pipeline, you can _optionally_ configure your Collector and Iglu Server to have an HTTPS-enabled endpoint. This might be required in some cases to track events on strictly SSL-only websites, as well as to enable first-party tracking (by putting the Collector endpoint on the same sub-domain as your website).

**AWS:**

1. Navigate to Amazon Certificate Manager (ACM) in the AWS Console
2. Request a public certificate from the ACM portal for the domain you want to host these endpoints under (e.g. for the Collector this might be `c.acme.com`) - make sure you are in the _same_ region as your pipeline

- `Fully qualified domain name` will be something like `c.acme.com`
- `Validation method` is whatever works best for you - generally `DNS validation` is going to be easiest to action
- `Key algorithm` should be left as `RSA 2048`

3. Once you have requested the certificate, it should show up in the ACM certificate list as `Pending validation` - complete the DNS / Email validation steps and wait until the status changes to `Issued`
4. Copy the issued certificate’s `ARN` and paste it into your `terraform.tfvars` file under `ssl_information.certificate_arn`
5. Change `ssl_information.enabled` to `true`
6. Apply the `iglu_server` / `pipeline` module as you have done previously to attach the certificate to the Load Balancer
7. Add a `CNAME` DNS record for your requested domain pointing to the AWS Load balancer (e.g. `c.acme.com` -> `<lb-identifier>.<region>.elb.amazonaws.com`)

- If you are using `Route53` for DNS record management, you can instead [setup an Alias route](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html) which can help circumvent certain `CNAME` cloaking tracking protections

You should now be able to access your service over HTTPS. Verify this by going to your newly set up endpoint in a browser — you should get a valid response with a valid SSL certificate.

**GCP:**

1. Navigate to Google Certificate Manager in the GCP Console
2. Select the `Classic Certificates` tab and then press `Create SSL Certificate`

- `Name` should be something unique for the certificate you are creating
- `Description` describe the service you are creating the certificate for
- `Create mode` allows you to either supply your own certificate or create a `Google-managed` one. We generally recommend the latter for ease of implementation and updates
- `Domains` add the domain you want to host the service under (e.g. for the Collector this might be `c.acme.com`)

3. Add an `A` DNS record for the requested domain pointing to the GCP Load balancer (e.g. `c.acme.com` -> `<lb-ipv4-address>`)
4. Copy the Certificate ID into your `terraform.tfvars` file under `ssl_information.certificate_id`

- The ID takes the form `projects/{{project}}/global/sslCertificates/{{name}}`

5. Change `ssl_information.enabled` to `true`
6. Apply the `iglu_server` / `pipeline` module as you have done previously to attach the certificate to the Load Balancer

Once the certificate is issued, you should now be able to access your service over HTTPS. Verify this by going to your newly set up endpoint in a browser — you should get a valid response with a valid SSL certificate.

> **Info:** It’s worth noting that GCP Managed Certificates can take up to 24 hours to be provisioned successfully, so it might take a while before you can access your service over HTTPS.

**Azure:**

Unlike AWS or GCP, Azure doesn't have a Certificate Management layer. So to follow this guide, you will need to have bought your own valid SSL certificate covering the domain(s) you want to bind to.

1. [Convert your SSL Certificate](https://github.com/snowplow-devops/terraform-azurerm-lb?tab=readme-ov-file#adding-a-custom-certificate) into the correct format (`pkcs12`) needed for the Azure Application Load Balancer
2. Copy the `pkcs12` certificate password into your `terraform.tfvars` file under `ssl_information.password`
3. Copy the `pkcs12` certificate into your `terraform.tfvars` file under `ssl_information.data`

- Ensure the certificate is `base64`-encoded (e.g. `cat cert.p12 | base64`)

4. Change `ssl_information.enabled` to `true`
5. Apply the `iglu_server` / `pipeline` module as you have done previously to attach the certificate to the Load Balancer
6. Add an `A` DNS record for your requested domain pointing to the Azure Load Balancer (e.g. `c.acme.com` -> `<lb-ipv4-address>`)

You should now be able to access your service over HTTPS. Verify this by going to your newly set up endpoint in a browser — you should get a valid response with a valid SSL certificate.

***

***

If you are curious, here’s [what has been deployed](/docs/get-started/self-hosted/what-is-deployed/). Now it’s time to [send your first events to your pipeline](/docs/get-started/tracking/)!

---

# Telemetry principles in Snowplow Self-Hosted
> Learn about telemetry data collection in Snowplow Self-Hosted applications, what data is collected, and how to disable it.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/telemetry/

Telemetry helps us better understand how our applications are used:

- Which applications, clouds and warehouses are more popular than others?
- What are the most common pipeline topologies?
- Are users successful in running our stack over long periods of time?
- And so on.

This data is important for us when deciding where to invest our efforts to build a better product for our users (including you!).

## What data is collected?

In general, we track:

- Heartbeat events that tell us Snowplow applications are alive.
- Events regarding installation, startup, shutdown, etc, of our Terraform modules and applications.
- Metadata such as application version, cloud and region.

You can always disable telemetry if you prefer.

## Our principles

### Privacy

We do not automatically collect any personally identifiable information (PII) other than the IP address of the computer where a Snowplow application or a terraform module is running. This IP address is subsequently pseudonymised using SHA-256 with the [Snowplow PII pseudonymization enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/).

### Minimalism

We only ever collect what is required at any given point in time. We do not pre-empt future requirements or collect anything “just in case”. We also make sure telemetry does not affect application performance in any way.

### Transparency

Not only is our telemetry code open source (e.g. this [terraform module](https://github.com/snowplow-devops/terraform-snowplow-telemetry)), you can also inspect the schema we use for our telemetry events [here](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

## How can I help?

It helps our product development immensely if you keep telemetry enabled. We promise to keep it anonymous and as minimal as possible!

We also appreciate if you provide your email (or just a UUID) in the `user_provided_id` (or `userProvidedId`) setting. This allows us to tie events together across resources and offers a more complete picture of how the pipeline has been orchestrated. If you do provide an email address, we will only ever contact you with exciting Product & Engineering updates and Research studies. You can always exercise your right to be forgotten by [contacting us](https://snowplow.io/contact-us/).

## Which components have telemetry?

At the moment, opt-out telemetry is present in the following:

- Terraform modules for the [quick start guide](/docs/get-started/self-hosted/quick-start/).
- [Collector](/docs/api-reference/stream-collector/setup/).
- Enrich ([Enrich Kinesis](/docs/api-reference/enrichment-components/enrich-kinesis/), [Enrich PubSub](/docs/api-reference/enrichment-components/enrich-pubsub/), [Enrich Kafka](/docs/api-reference/enrichment-components/enrich-kafka/).
- RDB Loader ([Transformer Kinesis](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/), [Transformer PubSub](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/), [Redshift Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/redshift-loader/), [Snowflake Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/snowflake-loader/), [Databricks Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/databricks-loader/)).
- Snowplow Mini for [AWS](/docs/api-reference/snowplow-mini/setup-guide-for-aws/) and [GCP](/docs/api-reference/snowplow-mini/setup-guide-for-gcp/).
- [Snowbridge](/docs/api-reference/snowbridge/getting-started/).
- [Lake loader](/docs/api-reference/loaders-storage-targets/lake-loader/).

See the telemetry notice for each component linked above for more details.

---

# Snowplow Self-Hosted upgrade guide
> Step-by-step guide to upgrading Snowplow Self-Hosted Community Edition components and Terraform modules to newer versions.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/upgrade-guide/

Now that you have successfully spun up a Snowplow pipeline, you might want to upgrade your components when there are updates available.

The Community Edition Quick Start is updated in line with the Snowplow Community Edition Distributions and here we describe the steps you need to take to upgrade between versions.

> **Note:** You may have deviated from the original Quick Start example Terraform files. If this is the case then the concepts described here still apply. However, you will want to compare the differences between Quick Start releases to see what you need to update.

## Updating your repository

To upgrade to a new Quick Start version, you first need to update the git repository to the next release. We recommend upgrading one release at a time to ease the transition rather than trying to leap from an older version to the latest, although this should also be possible with some caution.

You'll first need to commit or stash any changes you haven't commited. If you've updated your `.tfvars` then you should stash these first (use `git stash`) and once you've updated your repository, you can retrieve your changes from the stash (use `git stash pop`).

To jump to the latest release, you should update your local `main` branch to latest:

```bash
git checkout main

git pull
```

If you only want to move to a specific version then you should checkout that tag into a new branch.

```bash
git checkout -b 21.08 21.08 // To checkout 21.08
```

## Migrations

> **Info:** This information is currently specific to AWS.

Before updating your stack, you should run any required manual migrations. These are listed below.

If the specific upgrade path isn't mentioned below, then you should be able to upgrade without any manual interventation.

Clicking the links below will show all the Terraform differences between each version.

### 22.01 to 22.01-Patch.1

#### Changing the name of the `pipeline_db*` variables

Since we have two destination database options (`postgres` and `snowflake`) starting with this release, we've replaced 'pipeline\_db' part in the variable names with 'postgres\_db'. You need to change your variable names accordingly.

Also, new 'pipeline\_db' variable is introduced. This variable specifies the database that will be used as destination in the pipeline. It can be either 'postgres' or 'snowflake'. Since all existing pipelines are using Postgres if they aren't modified, this variable should be set to 'postgres' in those pipelines.

### 21.04 to 21.08

#### Changing some of the column types

The database schemas created with a 21.04 Quick Start do not work well with events enriched with pii pseudoanonymization. Therefore, a Postgres migration is required to change types of some of the columns. You need to apply [this migration script](https://github.com/snowplow-incubator/snowplow-postgres-loader/blob/master/migrations/0-3-0.sql) to your database (You may need to change the `public` schema to `atomic` for this to run successfully with the standard Quick Start installation).

#### Cloudwatch logs are on by default

21.08 Quick Start has enabled Cloud Watch logs by default. We recommend you leave this enabled as the cloud watch logs can be beneficial when diagnosing any issues with your pipeline components.

## Updating Snowplow components

If you are using a standard Quick Start deployment, you can follow the steps below to upgrade your Iglu Server and Pipeline. If you have deviated from the example, then you should compare the differences between the Snowplow Community Edition Distributions in the examples and update your usage of the Terraform modules accordingly.

> **Note:** You will experience some brief downtime when running these upgrades, as there is no redudency in the Community Edition.

#### Iglu Server

Now you are ready to update your Iglu Server. The example below is for the `default` configuration on `aws`. You should change to the correct folder for your preferred installation.

```bash
cd terraform/aws/iglu_server/default

terraform init

terraform plan
```

At this stage, you should see the changes that will be made to your infrastructure. If everything looks ok and there are no errors you can apply the update.

```bash
terraform apply
```

#### Pipeline

Now you are ready to update your pipeline components. The example below is for the `default` configuration on `aws`. You should change to the correct folder for your preferred installation.

```bash
cd terraform/aws/pipeline/default

terraform init

terraform plan
```

At this stage, you should see the changes that will be made to your infrastructure. If everything looks ok and there are no errors you can apply the update.

```bash
terraform apply
```

---

# What is deployed in the Self-Hosted Quick Start?
> Overview of Snowplow pipeline infrastructure components deployed by Terraform including collector, enrichment, and loader modules.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/what-is-deployed/

Let’s take a look at what is deployed when you follow the quick start guide.

> **Tip:** You can very easily edit the script or run each of the Terraform modules independently, giving you the flexibility to design the topology of your pipeline according to your needs.

## Overview

**AWS:**

**Redshift:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(EC2)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kinesis app</i>
(EC2)"}}
iglu{{"<b>Iglu Server</b>
(EC2)"}}
igludb[("<b>Iglu Database</b>
(RDS)")]
bad[["<b>Bad Stream</b>
(Kinesis)"]]
loader("<b>Redshift Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(Redshift)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kinesis)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kinesis)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Archival and failed events

```mermaid
flowchart LR
      good[["<b>Enriched Stream</b>
(Kinesis)"]]
        blobloadergood{{"<b>S3 Loader Good</b>
(EC2)"}}
        blobgood[("<b>Enriched Events</b>
(S3)")]
        good-->blobloadergood-->blobgood
        bad[["<b>Bad Stream</b>
(Kinesis)"]]
        blobloaderbad{{"<b>S3 Loader Bad</b>
(EC2)"}}
        blobbad[("<b>Failed Events</b>
(S3)")]
        bad-->blobloaderbad-->blobbad
```

#### Redshift Loader

For more information about the Redshift Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/?warehouse=redshift).

**Snowflake:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(EC2)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kinesis app</i>
(EC2)"}}
iglu{{"<b>Iglu Server</b>
(EC2)"}}
igludb[("<b>Iglu Database</b>
(RDS)")]
bad[["<b>Bad Stream</b>
(Kinesis)"]]
loader("<b>Snowflake Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(Snowflake)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kinesis)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kinesis)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Archival and failed events

```mermaid
flowchart LR
      good[["<b>Enriched Stream</b>
(Kinesis)"]]
        blobloadergood{{"<b>S3 Loader Good</b>
(EC2)"}}
        blobgood[("<b>Enriched Events</b>
(S3)")]
        good-->blobloadergood-->blobgood
        bad[["<b>Bad Stream</b>
(Kinesis)"]]
        blobloaderbad{{"<b>S3 Loader Bad</b>
(EC2)"}}
        blobbad[("<b>Failed Events</b>
(S3)")]
        bad-->blobloaderbad-->blobbad
```

#### Snowflake Loader

For more information about the Snowflake Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/).

**Databricks:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(EC2)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kinesis app</i>
(EC2)"}}
iglu{{"<b>Iglu Server</b>
(EC2)"}}
igludb[("<b>Iglu Database</b>
(RDS)")]
bad[["<b>Bad Stream</b>
(Kinesis)"]]
loader("<b>Databricks Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(Databricks)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kinesis)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kinesis)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Archival and failed events

```mermaid
flowchart LR
      good[["<b>Enriched Stream</b>
(Kinesis)"]]
        blobloadergood{{"<b>S3 Loader Good</b>
(EC2)"}}
        blobgood[("<b>Enriched Events</b>
(S3)")]
        good-->blobloadergood-->blobgood
        bad[["<b>Bad Stream</b>
(Kinesis)"]]
        blobloaderbad{{"<b>S3 Loader Bad</b>
(EC2)"}}
        blobbad[("<b>Failed Events</b>
(S3)")]
        bad-->blobloaderbad-->blobbad
```

#### Databricks Loader

For more information about the Databricks Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/?warehouse=databricks).

***

**GCP:**

**BigQuery:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(CE)"}}
enrich{{"<b>Enrich</b>
<i>enrich-pub/sub app</i>
(CE)"}}
iglu{{"<b>Iglu Server</b>
(CE)"}}
igludb[("<b>Iglu Database</b>
(CloudSQL)")]
bad[["<b>Bad Stream</b>
(Pub/Sub)"]]
loader("<b>BigQuery Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(BigQuery)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Pub/Sub)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Pub/Sub)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### BigQuery Loader

For more information about the BigQuery Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/bigquery-loader/).

***

**Azure:**

**Databricks:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(VMSS)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kafka app</i>
(VMSS)"}}
iglu{{"<b>Iglu Server</b>
(VMSS)"}}
igludb[("<b>Iglu Database</b>
(Postgres)")]
bad[["<b>Bad Stream</b>
(Kafka)"]]
loader{{"<b>Data Lake Loader</b>
(VMSS)"}}
atomic[("<b>Events</b>
(ADLS Gen2)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kafka)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kafka)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Data Lake Loader

For more information about the Data Lake Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/lake-loader/).

**Synapse Analytics:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(VMSS)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kafka app</i>
(VMSS)"}}
iglu{{"<b>Iglu Server</b>
(VMSS)"}}
igludb[("<b>Iglu Database</b>
(Postgres)")]
bad[["<b>Bad Stream</b>
(Kafka)"]]
loader{{"<b>Data Lake Loader</b>
(VMSS)"}}
atomic[("<b>Events</b>
(ADLS Gen2)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kafka)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kafka)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Data Lake Loader

For more information about the Data Lake Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/lake-loader/).

***

***

## Collector load balancer

This is an application load balancer for your inbound HTTP(S) traffic. Traffic is routed from the load balancer to the Collector instances.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/alb/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/lb/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/lb/azurerm/latest)).

## Collector

This is an application that receives raw Snowplow events over HTTP(S), serializes them, and then writes them to Kinesis (on AWS), Pub/Sub (on GCP) or Kafka / Event Hubs (on Azure). More details can be found [here](/docs/api-reference/stream-collector/).

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/collector-kinesis-ec2/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/collector-pubsub-ce/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/collector-event-hub-vmss/azurerm/latest)).

## Enrich

This is an application that reads the raw Snowplow events, validates them (including validation against [schemas](/docs/fundamentals/schemas/)), [enriches](/docs/pipeline/enrichments/) them and writes the enriched events to another stream. More details can be found [here](/docs/api-reference/enrichment-components/).

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/enrich-kinesis-ec2/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/enrich-pubsub-ce/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/enrich-event-hub-vmss/azurerm/latest)).

## Iglu

The Iglu stack allows you to manage [schemas](/docs/fundamentals/schemas/).

### Iglu load balancer

This load balances the inbound traffic and routes traffic to the Iglu Server.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/alb/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/lb/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/lb/azurerm/latest)).

### Iglu Server

The [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) serves requests for Iglu schemas stored in your schema registry.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/iglu-server-ec2/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/iglu-server-ce/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/iglu-server-vmss/azurerm/latest)).

### Iglu database

This is the Iglu Server database (RDS on AWS, CloudSQL on GCP and PostgreSQL on Azure) where the Iglu [schemas](/docs/fundamentals/schemas/) themselves are stored.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/rds/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/cloud-sql/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/postgresql-server/azurerm/latest)).

## Streams

The various streams (Kinesis on AWS, Pub/Sub on GCP and Kafka / Event Hubs on Azure) are a key component of ensuring a non-lossy pipeline, providing crucial back-up, as well as serving as a mechanism to drive real time use cases from the enriched stream.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/kinesis-stream/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/pubsub-topic/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/event-hub-namespace/azurerm/latest)).

**AWS only — DynamoDB**

On the first run of each of the applications consuming from Kinesis (e.g. Enrich), the Kinesis Connectors Library creates a DynamoDB table to keep track of what they have consumed from the stream so far. Each Kinesis consumer maintains its own checkpoint information.

The DynamoDB autoscaling module enables autoscaling for a target DynamoDB table. Note that there is a `kcl_write_max_capacity` variable which can be set to your expected RPS, but setting it high will of course incur more cost.

You can find further details in the [DynamoDB Terraform module](https://registry.terraform.io/modules/snowplow-devops/dynamodb-autoscaling/aws/latest).

### Raw stream

Collector payloads are written to the raw stream, before being picked up by the Enrich application.

### Enriched stream

Events that have been validated and enriched by the Enrich application are written to the enriched stream. Depending on your cloud and destination, different loaders pick up the data from this stream, as shown on the diagram [above](#overview).

> **Note:** The S3 loader (enriched) also reads from this enriched stream and writes to the enriched folder on S3.

### Bad 1 stream

This bad stream is for [failed events](/docs/fundamentals/failed-events/), which get created when the Collector, Enrich or various loader applications fail to process the data.

### Other streams

**AWS:**

The _Bad 2 stream_ is for failed events generated by the S3 loader as it tries to write from the Bad 1 stream to the bad folder on S3.

If you selected Redshift, Snowflake or Databricks as your destination, the loader will use an extra SQS stream internally as explained in the [reference documentation](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/?cloud=aws).

**GCP:**

If you selected BigQuery as your destination, the _Bad Rows_ stream will contain events that could not be inserted into BigQuery by the loader. This includes data that is not valid against its schema or that is somehow corrupted in a way that the loader cannot handle. In addition, the loader users a few streams internally, as explained in the [reference documentation](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/).

**Azure:**

No other streams.

***

## Archival and failed events

**AWS:**

Aside from your main destination, the data is written to S3 by the S3 Loader applications for archival and to deal with [failed events](/docs/fundamentals/failed-events/).

See the [S3 Loader](https://registry.terraform.io/modules/snowplow-devops/s3-loader-kinesis-ec2/aws/latest) and [S3](https://registry.terraform.io/modules/snowplow-devops/s3-bucket/aws/latest) Terraform modules for further details on the resources, default and required input variables, and outputs.

The following loaders and folders are available:

- Enriched loader, `enriched/`: enriched events, in GZipped blobs of [enriched TSV](/docs/pipeline/enriched-tsv-format/). Historically, this has been used as the staging ground for loading into data warehouses via the [Batch transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/) application. However, it’s no longer used in the quick start examples.
- Bad loader, `bad/`: [failed events](/docs/fundamentals/failed-events/). You can [query them using Athena](/docs/monitoring/exploring-failed-events/file-storage/).

**GCP:**

To store all failed events, you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/) application.

**Azure:**

Currently, [failed events](/docs/fundamentals/failed-events/) are only available in the `bad` Kafka / Event Hubs topic.

***

## Loaders

**Redshift:**

[RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is a set of applications that loads enriched events into Redshift.

See the following Terraform modules for further details on the resources, default and required input variables, and outputs:

- [Transformer Kinesis](https://registry.terraform.io/modules/snowplow-devops/transformer-kinesis-ec2/aws/latest)
- [Redshift Loader](https://registry.terraform.io/modules/snowplow-devops/redshift-loader-ec2/aws/latest)

**BigQuery:**

The [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/) is a set of applications that loads enriched events into BigQuery.

See the [Terraform module](https://registry.terraform.io/modules/snowplow-devops/bigquery-loader-pubsub-ce/google/latest) for further details on the resources, default and required input variables, and outputs.

There will be a new dataset available in BigQuery with the suffix `_snowplow_db`. Within this dataset, there will be a table called `events` — all of your collected events will be available here generally within a few seconds after they are sent into the pipeline.

**Snowflake:**

[Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) is an application that loads data into Snowflake.

See the following Terraform module for further details on the resources, default and required input variables, and outputs:

- Snowflake Streaming Loader ([AWS](https://registry.terraform.io/modules/snowplow-devops/snowflake-streaming-loader-ec2/aws/latest))

**Databricks (direct):**

[RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is a set of applications that loads enriched events into Databricks.

See the following Terraform modules for further details on the resources, default and required input variables, and outputs:

- [Transformer Kinesis](https://registry.terraform.io/modules/snowplow-devops/transformer-kinesis-ec2/aws/latest)
- [Databricks Loader](https://registry.terraform.io/modules/snowplow-devops/databricks-loader-ec2/aws/latest)

**Databricks (via lake):**

[Lake Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is an application that loads enriched events into a data lake so that they can be queried via Databricks (or other means).

See the Lake Loader [Terraform module](https://registry.terraform.io/modules/snowplow-devops/lake-loader-vmss/azurerm/latest) for further details on the resources, default and required input variables, and outputs.

The Terraform stack for the pipeline will deploy a storage account and a storage container where the loader will write the data.

**Synapse Analytics:**

[Lake Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is an application that loads enriched events into a data lake so that they can be queried via Synapse Analytics (or Fabric, OneLake, etc).

See the Lake Loader [Terraform module](https://registry.terraform.io/modules/snowplow-devops/lake-loader-vmss/azurerm/latest) for further details on the resources, default and required input variables, and outputs.

The Terraform stack for the pipeline will deploy a storage account and a storage container where the loader will write the data.

***

---

# Strong cookies and ad-blockers
> Configure first-party cookies and ad-blocker resilient tracking with Cookie Extension service and custom collector domains.
> Source: https://docs.snowplow.io/docs/get-started/tracking/cookies-and-ad-blockers/

This page details a few recommendations to set up your Snowplow tracking. Their goal is to make sure that:

1. Your Web app has strong cookies to support persisting the Snowplow user and session identifiers.
2. The Snowplow tracking is resilient to ad-blockers.

## 1. Strong cookies for user and session identifiers

**Strategy 1: First party cookies**

As browsers [cut down on third-party cookies](https://snowplow.io/blog/privacy-updates-ad-blockers-and-first-party-tracking/), they are becoming less and less useful for storing user and session identifiers. The alternative is to use first-party cookies. Although first-party cookies don't allow sharing user and session identifiers across domains (they do let you share them across sub-domains), there are other alternative solutions for [cross-domain tracking](/docs/sources/web-trackers/cross-domain-tracking/) that can be used in conjunction with first-party cookies.

Using Snowplow, you can set up your collector to be on the same domain as the website, which is the requirement for the use of first-party cookies. To learn how to set up the collector domain, [visit this page](/docs/sources/first-party-tracking/).

**Strategy 2: Snowplow Cookie Lifetime Extension service for Safari ITP**

As of Safari 16.4 released in April 2023, Safari sets the [lifetime of server-set cookies](https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense) to a maximum of 7 days under some circumstances even if they are first-party cookies. This greatly limits the effectiveness of tracking a customer journey where users are not regularly returning to your website. In particular, it affects the `network_userid` identifier in Snowplow events.

Snowplow provides the Cookie Lifetime Extension service solution that fully mitigates the impact of this change. Visit the [documentation for the Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) to learn more.

## 2. Mitigating the impact of ad-blockers

**Strategy 1: Custom POST path on the collector**

The recommended method for sending events to the Snowplow collector is using POST requests. These requests are by default made to the `/com.snowplowanalytics.snowplow/tp2` path on the collector. As many ad blockers will block this path, it is recommended to change the path.

To change the path, you will need to take the following steps (in this order):

1. Update the [Collector configuration](/docs/pipeline/collector/) to support the new path _before_ you send events to it with this setting.
2. Provide the new path in the [`customPostPath` configuration in the tracker](/docs/sources/web-trackers/configuring-how-events-sent/#custom-post-path).

**Strategy 2: Rename sp.js file**

In case you are using the Snowplow JavaScript tracker using the `sp.js` file that you include using an HTML tag, it is advisable to rename this file to reduce the chance of ad-blockers preventing the script from loading. You may rename the file to a random 8 character string, see the [recommendations here](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/self-hosting-the-javascript-tracker-aws/).

In case you have a JavaScript app (e.g., built using React, Angular, Vue.js), consider installing the tracker using the NPM package. This approach is more resilient against ad-blockers and advisable for single page apps. See the [installation instructions here](/docs/sources/web-trackers/tracker-setup/).

**Strategy 3: Server-side tracking**

Finally, the most robust solution against the impact of ad-blockers and other browser privacy measures is the use of server-side tracking. Through server-side tracking, you will track events directly from your server app (e.g., in Python, Node.js, Ruby). Server-side tracking works within first party context allowing you to build on top of the first-party cookies for user identification.

Snowplow provides tracker SDKs for all commonly used programming languages used in server-side apps (see the [full list here](/docs/sources/)). To learn more about server-side tracking using Snowplow, [visit this blog post](https://snowplow.io/blog/server-side-tracking-vs-client-side-tracking/).

---

# Tracking your first events
> Tracking your first Snowplow events
> Source: https://docs.snowplow.io/docs/get-started/tracking/

Once your pipeline is set up, you will want to send some events to it. Here's an overview of the different options.

> **Tip:** Regardless of how you send the events, it might take a few minutes for them to reach your destination (e.g. data warehouse). This depends on which [destination and loader](/docs/destinations/warehouses-lakes/) you have configured.

## A quick test

If you are eager to look at some Snowplow events or just test your pipeline, you can use the box below to send some data (including [failed events](/docs/fundamentals/failed-events/)) to your Collector.

> **Tip:** Note that to use this tool, you need a Collector URL that starts with `https://`, not `http://`. This is because web browsers block traffic from HTTPS-enabled sites (such as `https://docs.snowplow.io` — this site — to non-HTTPS resources).

**CDI Private Managed Cloud:**

You can find the Collector URL (Collector endpoint) in [Console](https://console.snowplowanalytics.com/environments).

**CDI Cloud:**

You can find the Collector URL (Collector endpoint) in [Console](https://console.snowplowanalytics.com/environments).

**Snowplow Self-Hosted:**

Input the Collector URL you chose when deploying your Snowplow Self-Hosted pipeline.

If you have not yet configured an SSL certificate and a custom domain name for your Collector, you can use `http://<collector_dns_name>` (`http`, not `https`), where `collector_dns_name` is the output of the pipeline Terraform module.

***

**Which events are sent?**

We use the following tracking code:

```javascript
window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: 'iglu:com.snowplowanalytics.snowplow/add_to_cart/jsonschema/1-0-0',

      data: { sku: 'ASO01043', unitPrice: 49.95, quantity: 1000 }

    }

  }

)

window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',

      data: {

        targetUrl: 'https://snowplow.io'

      }

    }

  }

)

window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: 'iglu:com.snowplowanalytics.mobile/deep_link/jsonschema/1-0-0',

      data: {

        url: 'https://snowplowanalytics.com'

      }

    }

  }

)

window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',

      data: {

        targetUrl: 'https://snowplow.io'

      }

    },

    context: [

      {

        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',

        data: {

          id: '6a2a8c7c-8e7a-4e9f-9b4c-0f4f9a3f35a9'

        }

      }

    ]

  }

)

window.snowplow('trackPageView', undefined)

window.snowplow('trackPageView', { title: 'My custom page title' })



// Bad Events

window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: 'iglu:com.fake_event/event/jsonschema/1-0-0',

      data: { foo: 'bar' }

    }

  }

)

window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: '',

      data: {}

    }

  }

)

window.snowplow('trackSelfDescribingEvent',

  {

    event: {

      schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',

      data: {

        invalidProperty: "This schema doesn't have this property"

      }

    }

  }

)
```

> **Tip:** An application ID is sent along with each Snowplow event. You can pick a value that will make it easy to filter out these test events from your data later, e.g. like this:
>
> ```sql
> ...
>
> WHERE app_id != 'test'
>
> ...
> ```

Now, let’s take a look at how to set up actual event tracking.

## Using the JavaScript tracker

The [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/) is our most commonly used tracker. It’s a good choice for websites, and the installation process is similar to other tools like Google Analytics.

To use the JavaScript tracker on your site, you will need to obtain a code snippet first.

**CDI Private Managed Cloud:**

CDI Private Managed Cloud can automatically generate the snippet for you. Go to the [tag generator](https://console.snowplowanalytics.com/tag-generator) screen, fill in the necessary parameters, and copy the snippet at the bottom.

**CDI Cloud:**

You can find the pre-generated snippet in the [Getting started](https://console.snowplowanalytics.com/) section.

**Snowplow Self-Hosted:**

Take note of the Collector URL you’ve chosen when deploying your Snowplow Self-Hosted pipeline.

If you have not yet configured an SSL certificate and a custom domain name for your Collector, you can use `http://<collector_dns_name>` (`http`, not `https`), where `collector_dns_name` is the output of the pipeline Terraform module.

Then, follow the JavaScript tracker [quick start guide](/docs/sources/web-trackers/quick-start-guide/) to create your snippet.

***

Once you have the snippet, there are two common ways to deploy it.

**Editing your website directly:**

If you have access to the source code of your site, paste the snippet into the `<head>` HTML section and deploy the changes.

**Using Google Tag Manager:**

If you are already using Google Tag Manager to add various code snippets to your site, you can add your Snowplow snippet there.

![Animated walkthrough of adding the Snowplow JavaScript snippet as a Custom HTML tag in Google Tag Manager, setting it to fire on All Pages, and previewing the tag](/assets/images/gtm-01e0ff314a61e447de24b8aa69cf7ee7.gif)

- Navigate to the Google Tag Manager account you wish to add tracking to
- Create a new Custom HTML tag and paste the Javascript snippet into the tag
- Set it to fire on 'All Pages' or a trigger of your choosing
- You can preview your tag to send some events before publishing it

***

The JavaScript tracker captures many events (e.g. page views) automatically, so you should start accumulating your first events as soon as the changes are rolled out to your users.

> **Tip:** To make sure that your tracking is well configured for strong first-party cookies and resilient against the impact of ad-blockers, [visit this page with several useful recommendations](/docs/get-started/tracking/cookies-and-ad-blockers/).

## Using other trackers

We have many different trackers in different programming languages, in case the JavaScript tracker is not a fit for you. For example, see the [mobile native trackers](/docs/sources/mobile-trackers/) or the [full list](/docs/sources/) of what’s available.

> **Tip:** For quick testing, you might be tempted to send data to your Collector URL using a basic command-line tool like `cURL`. However, you would need to ensure that the data format follows our [tracker protocol](/docs/events/). Instead, take a look at [sending events with Snowplow CLI](/docs/api-reference/snowplow-cli/send-events/), which will do this for you.

---

# Glossary
> Comprehensive glossary of Snowplow terms covering components, concepts, legal terms, and product offerings.
> Source: https://docs.snowplow.io/docs/glossary/

This page provides a glossary of terms used by Snowplow.

To help clarify, we've categorized the terms:

- **Component**: a pipeline application, package, SDK, or other code-based tool or set of tools, e.g. [Snowplow Micro](/docs/testing/snowplow-micro/)
- **Concept**: names for Snowplow things and ideas, e.g. [tracking plan](/docs/fundamentals/tracking-plans/)
- **Legal**: about licensing, e.g. [SLULA](/docs/licensing/)
- **Offering**: a Snowplow product that you can buy, e.g. [Snowplow CDI Cloud](/docs/get-started/) or [Snowplow Signals](/docs/signals/introduction/)

| Term                                   | Category  | Description                                                                                                                                                           | More information                                                                           |
| -------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| **Atomic events table**                | Concept   | Core warehouse table containing all Snowplow events                                                                                                                   | [Atomic events table](/docs/fundamentals/warehouse-tables/)                                |
| **Atomic fields**                      | Concept   | Core event properties defined by the Snowplow atomic schema                                                                                                           | [Out-of-the-box fields](/docs/fundamentals/canonical-event/#common-fields)                 |
| **Attribute**                          | Concept   | Individual data point or characteristic calculated by Signals                                                                                                         | [Attributes](/docs/signals/concepts/#types-of-attribute)                                   |
| **Attribute group**                    | Concept   | Versioned collection of related attributes that defines how Signals should calculate behavioral data                                                                  | [Attribute groups](/docs/signals/concepts/#attribute-groups)                               |
| **Attribute key**                      | Concept   | Event or entity property that defines the context in which attributes are calculated by Signals                                                                       | [Attribute groups](/docs/signals/concepts/#attribute-keys)                                 |
| **Collection**                         | Concept   | Generation of event data and sending to an event Collector endpoint                                                                                                   | [Sources](/docs/sources/)                                                                  |
| **Collector**                          | Component | Application that receives and saves raw events, and forwards them to Enrich                                                                                           | [Collector](/docs/pipeline/collector/)                                                     |
| **Context**                            | Concept   | Deprecated name for entity                                                                                                                                            | [Entities](/docs/fundamentals/entities/)                                                   |
| **Cookie Extension service**           | Concept   | Process for extending cookie lifetimes                                                                                                                                | [Cookie Extension](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) |
| **Data contract**                      | Concept   | Event definitions; set of tracking plans                                                                                                                              | [Event Studio](/docs/event-studio/)                                                        |
| **Data model**                         | Component | dbt SQL package for processing events in the warehouse                                                                                                                | [Modeling your data](/docs/modeling-your-data/)                                            |
| **Data pipeline**                      | Concept   | Set of applications for collecting, processing, and storing events                                                                                                    | [Fundamentals](/docs/fundamentals/)                                                        |
| **Data product**                       | Concept   | Deprecated name for tracking plan                                                                                                                                     | [Tracking plans](/docs/event-studio/tracking-plans/)                                       |
| **Data Product Studio**                | Offering  | Former name for Event Studio                                                                                                                                          | [Event Studio](/docs/event-studio/)                                                        |
| **Data quality**                       | Concept   | Tooling and reporting related to failed events and QA                                                                                                                 | [Data quality](/docs/testing/)                                                             |
| **Data structure**                     | Concept   | Schema, plus metadata                                                                                                                                                 | [Data structures](/docs/event-studio/data-structures/)                                     |
| **Derived events table**               | Concept   | Warehouse table of events generated by a data model                                                                                                                   | [Modeling your data with dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/)       |
| **Destination**                        | Concept   | Third-party data store or data activation platform that can receive Snowplow event data                                                                               | [Destinations](/docs/fundamentals/destinations/)                                           |
| **Enrich**                             | Component | Application that validates and enriches events                                                                                                                        | [Enrichments](/docs/pipeline/enrichments/)                                                 |
| **Enrichment**                         | Component | A process, run by Enrich, that adds to or modifies event data                                                                                                         | [Enrichments](/docs/pipeline/enrichments/)                                                 |
| **Entity**                             | Concept   | Supplemental or additional data included in an event; a schema or data structure describing additional data                                                           | [Entities](/docs/fundamentals/entities/)                                                   |
| **Event**                              | Concept   | A payload, processed by a data pipeline, describing something that happened; the "main" schema or data structure (i.e. not an entity) in an event                     | [Events](/docs/fundamentals/events/)                                                       |
| **Event specification**                | Concept   | Set of data structures, including event and entities, that an event payload should match                                                                              | [Event specifications](/docs/event-studio/tracking-plans/event-specifications/)            |
| **Event Studio**                       | Offering  | Tracking and data contracts tooling                                                                                                                                   | [Event Studio](/docs/event-studio/)                                                        |
| **Event, canonical**                   | Concept   | One of the original Snowplow events - page view, page ping, or the legacy ecommerce transaction events - that has contains data for some atomic fields, but no schema | [Canonical events](/docs/fundamentals/canonical-event/)                                    |
| **Event, enriched**                    | Concept   | An event payload that's valid and has been successfully enriched                                                                                                      | [Enrichments](/docs/pipeline/enrichments/)                                                 |
| **Event, failed**                      | Concept   | An event payload that the pipeline had some problem processing                                                                                                        | [Failed events](/docs/fundamentals/failed-events/)                                         |
| **Event, self-describing**             | Concept   | Custom event, based on a custom schema                                                                                                                                | [Self-describing events](/docs/fundamentals/events/#self-describing-events)                |
| **Event, structured**                  | Concept   | Legacy, proto-custom event                                                                                                                                            | [Structured events](/docs/fundamentals/canonical-event/#structured-events)                 |
| **Event, unstructured**                | Concept   | Deprecated name for self-describing event                                                                                                                             | [Self-describing events](/docs/fundamentals/events/#self-describing-events)                |
| **Event, valid**                       | Concept   | An event payload that matches its schema definitions; an event that the pipeline has collected, validated, and enriched without failure (i.e. not a failed event)     | [Failed events](/docs/fundamentals/failed-events/)                                         |
| **Extension**                          | Offering  | Third-party integration                                                                                                                                               |                                                                                            |
| **Identities**                         | Offering  | Real-time identity resolution service                                                                                                                                 | [Identities](/docs/identities/)                                                            |
| **ID service**                         | Concept   | Deprecated name for Cookie Extension service                                                                                                                          | [Cookie Extension](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) |
| **Iglu**                               | Component | Schema registry and associated tooling                                                                                                                                | [Iglu](/docs/api-reference/iglu/)                                                          |
| **Intervention**                       | Concept   | Signals user behavior trigger                                                                                                                                         | [Interventions](/docs/signals/concepts/#interventions)                                     |
| **License, Community**                 | Legal     | Deprecated license for certain Snowplow components; SCL                                                                                                               | [Copyright licenses](/docs/licensing/)                                                     |
| **License, Limited Use**               | Legal     | Non-production use license for certain Snowplow components; SLULA                                                                                                     | [Copyright licenses](/docs/licensing/)                                                     |
| **License, Personal and Academic**     | Legal     | Personal and academic license for certain Snowplow components; SPAL                                                                                                   | [Copyright licenses](/docs/licensing/)                                                     |
| **Loader**                             | Component | Application that loads data into warehouse or lake storage                                                                                                            | [Warehouses and lakes](/docs/destinations/warehouses-lakes/)                               |
| **Organization**                       | Concept   | Your organization, in the context of account management and SSO                                                                                                       | [Account management](/docs/account-management/)                                            |
| **Out-of-the-box**                     | Concept   | Non-custom events, tracking plans, schemas, etc. that are included with Snowplow                                                                                      |                                                                                            |
| **Profiles Store**                     | Component | Centralized Signals database that stores calculated attribute values                                                                                                  | [Profiles Store](/docs/signals/concepts/#profiles-store)                                   |
| **Registry**                           | Concept   | Server that stores schemas (schema repository)                                                                                                                        | [Iglu repositories](/docs/api-reference/iglu/iglu-repositories/)                           |
| **Reverse ETL**                        | Concept   | Sync insights from warehouse to other tools                                                                                                                           | [Reverse ETL](/docs/destinations/reverse-etl/)                                             |
| **Schema**                             | Concept   | JSON Schema that describes how data should look                                                                                                                       | [Schemas](/docs/fundamentals/schemas/)                                                     |
| **Schema respository**                 | Concept   | Server that stores schemas (schema registry)                                                                                                                          | [Iglu repositories](/docs/api-reference/iglu/iglu-repositories/)                           |
| **Service**                            | Concept   | Attribute consumption interface layer for Signals                                                                                                                     | [Services](/docs/signals/concepts/#services)                                               |
| **Signals**                            | Offering  | Real-time customer intelligence platform                                                                                                                              | [Signals](/docs/signals/)                                                                  |
| **SLULA**                              | Legal     | Snowplow Limited Use License Agreement, a non-production use license                                                                                                  | [Copyright licenses](/docs/licensing/)                                                     |
| **Snowbridge**                         | Component | Application that sends data streams into external destinations                                                                                                        | [Snowbridge](/docs/api-reference/snowbridge/)                                              |
| **Snowplow BDP**                       | Offering  | Deprecated name for Snowplow CDI                                                                                                                                      | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow BDP Enterprise**            | Offering  | Deprecated name for Snowplow CDI Private Managed Cloud                                                                                                                | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CDI**                       | Offering  | Full Snowplow product offering                                                                                                                                        | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CDI Cloud**                 | Offering  | Snowplow running in a cloud hosted by Snowplow                                                                                                                        | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CDI Private Managed Cloud** | Offering  | Snowplow running in your own cloud                                                                                                                                    | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CLI**                       | Component | Tool for command-line management of tracking plans and data structures                                                                                                | [Snowplow CLI](/docs/api-reference/snowplow-cli/)                                          |
| **Snowplow Community Edition**         | Offering  | Limited license (SLULA) Snowplow self-hosted product offering                                                                                                         | [Snowplow Self-Hosted](/docs/get-started/#self-hosted)                                     |
| **Snowplow Console**                   | Offering  | Web UI for Snowplow data management                                                                                                                                   | [Snowplow Console](https://console.snowplowanalytics.com)                                  |
| **Snowplow ID**                        | Concept   | Unified identifier added to events by Identities                                                                                                                      | [Identities concepts](/docs/identities/concepts/)                                          |
| **Snowplow Inspector**                 | Component | Browser extension for testing and validation of web tracking                                                                                                          | [Snowplow Inspector](/docs/testing/snowplow-inspector/)                                    |
| **Snowplow Micro**                     | Component | Testing and QA pipeline                                                                                                                                               | [Snowplow Micro](/docs/testing/snowplow-micro/)                                            |
| **Snowplow Mini**                      | Component | Legacy testing and QA pipeline (see Snowplow Micro)                                                                                                                   | [Snowplow Mini](/docs/api-reference/snowplow-mini/)                                        |
| **Snowplow Self-Hosted**               | Offering  | Production self-hosted Snowplow product offering                                                                                                                      | [Snowplow Self-Hosted](/docs/get-started/#self-hosted)                                     |
| **Snowtype**                           | Component | Tool for custom tracking code generation                                                                                                                              | [Snowtype](/docs/event-studio/implement-tracking/)                                         |
| **Snowtype CLI**                       | Component | Tool for command-line management of Snowtype code generation                                                                                                          | [Snowtype CLI](/docs/event-studio/implement-tracking/install-snowtype/)                    |
| **Source**                             | Concept   | Tracker SDK or webhook that sends events to a Collector endpoint                                                                                                      | [Sources](/docs/sources/)                                                                  |
| **Source application**                 | Concept   | Your application where you have implemented tracking                                                                                                                  | [Source applications](/docs/event-studio/source-applications/)                             |
| **Tracker**                            | Component | SDK that sends events to an event Collector endpoint                                                                                                                  | [Trackers](/docs/sources/)                                                                 |
| **Tracking catalog**                   | Concept   | Deprecated list of received event types in Snowplow Console                                                                                                           |                                                                                            |
| **Tracking plan**                      | Concept   | Set of event specifications, plus metadata                                                                                                                            | [Tracking plans](/docs/event-studio/tracking-plans/)                                       |
| **TSV**                                | Concept   | Tab-separated values format, used for events within a Snowplow pipeline                                                                                               | [Enriched TSV format](/docs/pipeline/enriched-tsv-format/#tsv-header)                      |
| **Validation**                         | Concept   | Pipeline process of checking events against their schemas                                                                                                             |                                                                                            |
| **Visualization**                      | Offering  | Accompanying dashboard for a data model                                                                                                                               | [Visualizations](/docs/modeling-your-data/visualization/)                                  |
| **Webhook**                            | Component | API that allows you to send external data to Snowplow pipelines, usually third-party                                                                                  | [Webhooks](/docs/sources/webhooks/)                                                        |
| **Workspace**                          | Concept   | Isolated environment containing one production pipeline, and potentially multiple QA or development pipelines                                                         |                                                                                            |

---

# Cross-domain tracking
> How Snowplow Identities resolves user identity across sites with different cookie domains using cross-domain tracking.
> Source: https://docs.snowplow.io/docs/identities/concepts/cross-domain-tracking/

When users navigate between sites with different cookie domains, or from a mobile app to a webview, each destination assigns its own `domain_userid`. Without cross-domain tracking, these appear as separate users. [Cross-domain](/docs/events/cross-navigation/), or cross-navigation, tracking solves this by passing the `domain_userid` from the source site or app in the URL. Events captured on the destination contain a `refr_domain_userid` field with the source's `domain_userid`.

This works for web-to-web navigation, mobile app-to-webview transitions, and any other scenario where the [web](/docs/sources/web-trackers/cross-domain-tracking/) or [native mobile](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) trackers decorate outgoing links.

You can [enable cross-domain tracking support](/docs/identities/configuration/#enable-cross-domain-tracking-aliases) in the Identities configuration so that Identities automatically extracts `refr_domain_userid` and maps it to `domain_userid` and `client_session_user_id`, linking the user's Snowplow IDs across sites.

## Example cross-domain resolution

In this example, ExampleCompany runs two sites on separate cookie domains: `brandA.com` and `brandB.com`. Cross-domain tracking is configured on the web tracker, and the ExampleCompany team has enabled cross-domain tracking in the Identities configuration.

A user browses `brandA.com` anonymously. The event contains a `domain_userid`. Identities creates a new Snowplow ID.

| Event property  | Value        |
| --------------- | ------------ |
| `url`           | `brandA.com` |
| `domain_userid` | `b0258d-...` |
| `user_id`       | -            |

```mermaid
graph TD
    A(["domain_userid:<br/>b0258d-..."])
    P1(("**Snowplow ID:<br/>sp_001**"))

    A --- P1
```

The user clicks a link to `brandB.com`. The destination site assigns a new `domain_userid`, but the event also contains a `refr_domain_userid` field with the `domain_userid` from `brandA.com`. Identities treats `refr_domain_userid` as equivalent to `domain_userid`, finds the existing Snowplow ID, and links the new `domain_userid` to it.

| Event property       | Value        |
| -------------------- | ------------ |
| `url`                | `brandB.com` |
| `domain_userid`      | `dd8ef0-...` |
| `refr_domain_userid` | `b0258d-...` |
| `user_id`            | -            |

```mermaid
graph TD
    A(["domain_userid:<br/>b0258d-..."])
    D(["domain_userid:<br/>dd8ef0-..."])
    P1(("**Snowplow ID:<br/>sp_001**"))

    A --- P1
    D --- P1
```

The user then logs into `brandB.com`. The event contains the same `domain_userid` from that site plus a `user_id`. Identities adds the `user_id` to the existing Snowplow ID.

| Event property  | Value                     |
| --------------- | ------------------------- |
| `url`           | `brandB.com/user-profile` |
| `domain_userid` | `dd8ef0-...`              |
| `user_id`       | `charlie@example.com`     |

```mermaid
graph TD
    A(["domain_userid:<br/>b0258d-..."])
    D(["domain_userid:<br/>dd8ef0-..."])
    U(["user_id:<br/>charlie@example.com"])
    P1(("**Snowplow ID:<br/>sp_001**"))

    A --- P1
    D --- P1
    U --- P1
```

All of the user's activity across both sites, both anonymous and authenticated, is resolved to the same Snowplow ID.

---

# Identities concepts
> Core concepts behind Snowplow Identities: identifiers, Snowplow IDs, merges, and identity resolution.
> Source: https://docs.snowplow.io/docs/identities/concepts/

Identities is based on several core concepts.

- **Identifiers** are the properties in the event payload that correspond to a user
- **A Snowplow ID** is a persistent, immutable identifier for a collection of linked identifiers that represent a single user
- **[Merges](/docs/identities/concepts/merges/)** occur when identifiers link two previously separate Snowplow IDs together
- **[Unique identifiers](/docs/identities/concepts/unique-identifiers/)** prevent incorrect merges when users share devices
- **[Cross-domain tracking](/docs/identities/concepts/cross-domain-tracking/)** resolves identity across sites with different cookie domains

## Identifiers

Identifiers are the properties in the event payload that Identities uses to resolve identity. They're key-value pairs. Each identifier has a type key, such as `domain_userid`, and a value that's the actual ID in the event payload, such as `a43eb2f1-...`.

You can configure identifiers from atomic event fields, or from a property inside an event or entity data structure. For example, you might add a hashed email address from a form submission event, or a global user ID from a custom user profile entity.

## Snowplow IDs

A Snowplow ID is a persistent, immutable identifier for a collection of linked identifiers that represent a single user.

When Identities receives a new identifier not found in another Snowplow ID, it creates a new Snowplow ID and links the identifier to it. The pipeline adds the Snowplow ID to the event in an [identity entity](#identity-entity).

In subsequent events, if the original identifier appears alongside new identifiers or identifier values, those are also linked to the same Snowplow ID. Events containing one or more of the linked identifiers will receive the same Snowplow ID.

### Example Snowplow ID creation

In this example, a user browses an ExampleCompany website anonymously, then logs in. Identities, running in the ExampleCompany Snowplow pipeline, is able to identify the events as belonging to the same user.

The configured identifiers for this example are:

- `domain_userid`: browser cookie ID from the web tracker
- `user_id`: the authenticated user's email address

During the anonymous browsing session, the only configured identifier in the events is the `domain_userid`. Since this is a new identifier, Identities creates a new Snowplow ID and links the identifier to it.

| Event property  | Value        |
| --------------- | ------------ |
| `domain_userid` | `4b0dfa-...` |
| `user_id`       | -            |

```mermaid
graph TD
    A(["domain_userid:<br/>4b0dfa-..."])
    B(("**Snowplow ID:<br/>sp_001**"))

    A --- B
```

After the user logs in, the next event contains both the `domain_userid` and their `user_id`. Identities finds the existing Snowplow ID via the `domain_userid` and adds the `user_id` to it.

| Event property  | Value               |
| --------------- | ------------------- |
| `domain_userid` | `4b0dfa-...`        |
| `user_id`       | `alice@company.com` |

```mermaid
graph TD
    A(["domain_userid:<br/>4b0dfa-..."])
    B(["user_id:<br/>alice@company.com"])
    C(("**Snowplow ID:<br/>sp_001**"))

    A --- C
    B --- C
```

All events associated with this user both pre- and post-login will have the same Snowplow ID `sp_001` attached in an identity entity.

## Identity resolution process

Identities stores the relationships between identifiers and Snowplow IDs using a graph-based model. This graph is the source of truth for identity resolution. It dynamically links and merges Snowplow IDs as new identifiers appear.

The identity resolution process for each event is as follows:

1. The Snowplow pipeline extracts the configured identifiers from the event payload

2. The pipeline sends the identifiers to the Identities service

3. The Identities service checks the graph for existing Snowplow IDs linked to the identifiers

4. Are any of the identifiers linked to existing Snowplow IDs?

   - No: Identities creates a new Snowplow ID, links all identifiers to it, and returns it
   - Yes: processing continues

5. Identities creates links to that Snowplow ID for any identifiers that aren't already linked

6. Do any of the identifiers have another linked Snowplow ID?

   - No: Identities returns the existing Snowplow ID
   - Yes: processing continues

7. Identities creates a merge relationship from the newer Snowplow ID to the older one

8. Identities returns the parent Snowplow ID and emits a merge event into the Snowplow pipeline to signal to downstream systems that a merge has occurred.

Identities resolves identity in real time. Each event is resolved against the current state of the graph, ensuring that the Snowplow ID reflects the most up-to-date identity information.

If the graph database is temporarily unavailable or slow to respond, Identities generates a deterministic fallback Snowplow ID linked to the event's identifiers, without looking up existing Snowplow IDs. Full identity resolution is deferred to avoid additional pipeline latency. An automatic background reconciliation process later replays the affected events to update the graph and generate any necessary merge events.

Tracked events are sometimes delayed in arriving into your pipeline, for example due to network issues or offline tracking. When they're eventually processed, they're resolved against the current state of the identity graph.

### Fallback IDs and identifier priority

Each identifier has a priority that determines how the Snowplow ID is generated when Identities can't reach the graph database, e.g. during a traffic spike. Higher-priority identifiers are preferred when generating fallback IDs.

In normal operation, priority doesn't affect how Snowplow IDs are linked. All identifiers contribute equally to identity resolution.

## Identities and Signals

Identities is designed to work alongside [Snowplow Signals](/docs/signals/get-started/). To use the Snowplow ID as a unified identifier in Signals, set an attribute group's key to `snowplow_id` and map it to the `snowplow_id` property in the identity entity.

## Data types

Identities adds two data types to your enriched event stream: an identity entity attached to every resolved event, and a merge event emitted when Snowplow IDs are combined.

### Identity entity

When Identities resolves identity for an event, it attaches an identity [entity](/docs/fundamentals/entities/) to the event payload. This entity contains the Snowplow ID that the event was resolved to. For [most warehouses](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/), you'll find it as `contexts_com_snowplowanalytics_snowplow_identity_2`.

### `identity`

**Type:** Entity

Identity entity that is enriched onto events by Snowplow Identities

**Properties:**

| Property               | Description |
| ---------------------- | ----------- |
| `created_at` _string_  | _Required._ |
| `snowplow_id` _string_ | _Required._ |

### Merge events

When a merge occurs, Identities emits a [self-describing event](/docs/fundamentals/events/#self-describing-events) into the enriched event stream. Downstream consumers such as the [Identities dbt package](/docs/identities/data-models/) use these events to keep identity mappings up to date.

The `merges` array contains **all** Snowplow IDs that have ever been merged into the current `snowplow_id`, not just the IDs involved in the most recent merge. This gives downstream consumers the complete merge history for a given Snowplow ID in a single event.

### `identity_merge`

**Type:** Event

Event emitted when Snowplow Identities merges user identities

**Properties:**

| Property               | Description                                       |
| ---------------------- | ------------------------------------------------- |
| `merged` _array_       | _Required._ Detailed info about merged identities |
| `merges` _array_       | _Required._ List of merged identity IDs           |
| `created_at` _string_  | _Required._ When the parent identity was created  |
| `snowplow_id` _string_ | _Required._ The parent/surviving identity ID      |

**Example warehouse query (Snowflake):**

```sql
select

unstruct_event_com_snowplowanalytics_snowplow_identity_merge_2 identity_merge_2

from

atomic.events

where

events.collector_tstamp > getdate() - interval '1 hour'

and events.event = 'unstruct'

and events.event_name = 'identity_merge'

and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Merges
> How Snowplow Identities merges Snowplow IDs when identifiers link previously separate users.
> Source: https://docs.snowplow.io/docs/identities/concepts/merges/

Merges happen when an event contains identifiers that are linked to different Snowplow IDs. This typically occurs when a user's anonymous activity is later connected to their known identity.

When Snowplow IDs merge, the older Snowplow ID becomes the ID for the combined set of identifiers. All identifiers from both Snowplow IDs are linked to the combined Snowplow ID. Merged Snowplow IDs can also be merged again in the future if new connecting identifiers are observed.

When a merge occurs, Identities emits a [merge event](/docs/identities/concepts/#merge-events) into your enriched event stream.

## Example merge process

In this example, a user has been browsing the ExampleCompany website. Identities, running in the ExampleCompany Snowplow pipeline, has created Snowplow ID `sp_001` with two linked identifiers: a `domain_userid` from the browser cookie, and a `user_id` from their authenticated email.

```mermaid
graph LR
    A(["domain_userid:<br/>4b0dfa-..."])
    B(["user_id:<br/>alice@company.com"])
    C(("**Snowplow ID:<br/> sp_001**"))

    A --- C
    B --- C
```

The same user now installs the ExampleCompany mobile app on their iPhone, and uses the app anonymously, i.e. they're not logged in.

The ExampleCompany team has configured Identities to use the `apple_idfv` identifier from the mobile platform entity.

Identities finds a new `apple_idfv` value in the user's first event, so it creates a new Snowplow ID.

| Event property | Value        |
| -------------- | ------------ |
| `apple_idfv`   | `6d920c-...` |
| `user_id`      | -            |

```mermaid
graph LR
    C(["apple_idfv:<br/>6d920c-..."])
    P2(("**Snowplow ID:<br/> sp_002**"))

    C --- P2
```

The user then logs into the mobile app. The next event contains the known `apple_idfv` and the previously seen `user_id`. Identities detects that `sp_001` and `sp_002` refer to the same user because of the matching `user_id`. It merges them and emits a merge event.

| Event property | Value               |
| -------------- | ------------------- |
| `apple_idfv`   | `6d920c-...`        |
| `user_id`      | `alice@company.com` |

The older Snowplow ID becomes the active one. All identifiers from both Snowplow IDs are now linked to `sp_001`; all future events containing any of these identifiers will have an identity entity containing `sp_001` attached.

```mermaid
graph LR
    A(["domain_userid:<br/>4b0dfa-..."])
    B(["user_id:<br/>alice@company.com"])
    C(["apple_idfv:<br/>6d920c-..."])
    P1(("**Snowplow ID:<br/> sp_001**"))
    P2(("**Snowplow ID:<br/> sp_002**"))

    A --- P1
    B --- P1
    C --- P2
    B --- P2
    P2 -. merged .-> P1
```

---

# Unique identifiers and merge conflicts
> How Snowplow Identities handles merge conflicts using unique identifiers to prevent incorrect merges.
> Source: https://docs.snowplow.io/docs/identities/concepts/unique-identifiers/

**Unique identifiers** are identifiers that should never cause two Snowplow IDs to merge together if they have different values. For example, if you mark `user_id` as unique, two events with different `user_id` values will never cause their Snowplow IDs to merge, even if they share other identifiers like `domain_userid`. This prevents incorrect merges such as when multiple users share a device or browser.

When Identities processes an event that would cause a merge, it checks whether any unique identifiers would conflict. If they do, Identities creates a new anonymous Snowplow ID for any identifiers that cannot be definitively attributed to either existing Snowplow ID.

## Example merge conflict

In this example, Alice has been using ExampleCompany's website and mobile app. Identities, running in the ExampleCompany Snowplow pipeline, has previously [merged](/docs/identities/concepts/merges/) her web and mobile Snowplow IDs into `sp_001`, linked to her `domain_userid`, `user_id`, and `apple_idfv` identifiers.

ExampleCompany have configured the `user_id` identifier as unique.

```mermaid
graph LR
    A(["domain_userid:<br/>4b0dfa-..."])
    B(["user_id:<br/>alice@company.com"])
    C1(["apple_idfv:<br/>6d920c-..."])
    P1(("**Snowplow ID:<br/> sp_001**"))
    P2(("**Snowplow ID:<br/> sp_002**"))

    A --- P1
    B --- P1
    C1 --- P2
    B --- P2
    P2 -. merged .-> P1
```

Alice shares her phone with a colleague, Bob. Bob logs into the ExampleCompany mobile app on Alice's phone. The event contains the device's `apple_idfv`, already linked to Alice's Snowplow ID via the earlier merge, and Bob's `user_id`.

| Event property | Value             |
| -------------- | ----------------- |
| `apple_idfv`   | `6d920c-...`      |
| `user_id`      | `bob@company.com` |

Identities detects a conflict: the `apple_idfv` is linked to a Snowplow ID with `user_id: alice@company.com`, but the event contains `user_id: bob@company.com`. Because `user_id` is marked as unique, Identities doesn't merge them. Instead, it creates a new Snowplow ID for Bob and links all of the event's identifiers to it — including the shared `apple_idfv`.

```mermaid
graph LR
    A(["domain_userid:<br/>4b0dfa-..."])
    B(["user_id:<br/>alice@company.com"])
    C1(["apple_idfv:<br/>6d920c-..."])
    P1(("**Snowplow ID:<br/> sp_001**"))
    P2(("**Snowplow ID:<br/> sp_002**"))

    A --- P1
    B --- P1
    C1 --- P2
    B --- P2
    P2 -. merged .-> P1

    C2(["apple_idfv:<br/>6d920c-..."])
    BobB(["user_id:<br/>bob@company.com"])
    BobC(("**Snowplow ID:<br/> sp_003**"))

    C2 --- BobC
    BobB --- BobC
```

Bob logs out, and a third person picks up the device and opens the app without logging in. The event contains only the `apple_idfv`. Identities looks up the identifier and finds it linked to Snowplow IDs with two different unique `user_id` values: `alice@company.com` and `bob@company.com`.

| Event property | Value        |
| -------------- | ------------ |
| `apple_idfv`   | `6d920c-...` |
| `user_id`      | -            |

Identities can't attribute this anonymous activity to either Alice or Bob. Instead of guessing, it creates a new anonymous Snowplow ID. The anonymous Snowplow ID is deterministic, so all future anonymous events from this device receive the same ID until a user logs in.

```mermaid
graph LR
    A(["domain_userid:<br/>4b0dfa-..."])
    B(["user_id:<br/>alice@company.com"])
    C1(["apple_idfv:<br/>6d920c-..."])
    P1(("**Snowplow ID:<br/> sp_001**"))
    P2(("**Snowplow ID:<br/> sp_002**"))

    A --- P1
    B --- P1
    C1 --- P2
    B --- P2
    P2 -. merged .-> P1

    C2(["apple_idfv:<br/>6d920c-..."])
    BobB(["user_id:<br/>bob@company.com"])
    BobC(("**Snowplow ID:<br/>sp_003**"))

    C2 --- BobC
    BobB --- BobC

    C3(["apple_idfv:<br/>6d920c-..."])
    AnonP(("**Snowplow ID:<br/>sp_004**<br/>_anonymous_"))

    C3 --- AnonP
```

> **Tip:** For web browsers, consider calling [`newSession`](/docs/sources/web-trackers/tracking-events/session/) or [`clearUserData`](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) when a user logs out. This resets the `domain_userid` cookie, preventing it from being shared between users on the same browser.

---

# Choosing identifiers for Identities
> How to choose stable identifiers for Snowplow Identities and validate their cardinality against your warehouse before configuring them.
> Source: https://docs.snowplow.io/docs/identities/configuration/choosing-identifiers/

The quality of identity resolution, and the cost and latency of running Identities, both depend on which event fields you configure as [identifiers](/docs/identities/concepts/#identifiers). Before adding an identifier in [Console](/docs/identities/configuration/), choose fields that are stable for each user, and validate their cardinality against your warehouse.

## What makes a good identifier

A good identifier holds the same value for a user over time and across their events. Stable identifiers let Identities link activity to a single [Snowplow ID](/docs/identities/concepts/#snowplow-ids) without creating unnecessary nodes in the identity graph.

Stable identifiers include:

- `user_id`, set when a user authenticates
- `domain_userid`, the web tracker's persistent cookie ID
- A hashed email address captured at sign-in
- A device identifier such as `apple_idfv`

Avoid identifiers whose value changes frequently for the same user. Each distinct identifier value becomes a node in the [identity graph database](/docs/identities/#how-identities-fits-into-the-snowplow-pipeline), so a value that changes often produces a new node on almost every event. Unstable identifiers include:

- Session identifiers such as `domain_sessionid`
- Page view identifiers
- Event identifiers such as `event_id`
- Any value that embeds a timestamp or is regenerated per session or per event

## Why high-cardinality identifiers are a problem

Identities stores every distinct identifier value as a node in a graph database. When you configure a high-cardinality identifier, the number of nodes grows roughly in line with your event volume rather than your user count.

This has three effects:

- Storage cost increases, because the graph database holds far more nodes than there are users
- Identity resolution latency increases, because graph operations run against a much larger graph
- Resolution quality drops, because most nodes are one-off values that never link any events together

An unstable identifier can grow the identity graph by orders of magnitude. Removing an identifier later doesn't reduce the nodes already created, and [affects resolution for all future events](/docs/identities/configuration/).

## Validate identifiers against your warehouse

It is recommended to run these checks against your `atomic.events` table before configuring an identifier. The examples use standard atomic columns. If your identifier comes from a custom event or entity field, replace the column with the relevant `unstruct_event_*` or `contexts_*` column. See [how data structures map to warehouse columns](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

### Check cardinality

Compare the number of distinct values for each candidate identifier against the total number of events. An identifier whose distinct count approaches the event count produces a new value on nearly every event, and isn't suitable.

```sql
select

  count(*)                       as total_events,

  count(distinct user_id)        as distinct_user_id,

  count(distinct domain_userid)  as distinct_domain_userid

from atomic.events

where collector_tstamp > current_date - interval '30 days';
```

### Check identifiers per unique identifier

For each candidate identifier, measure how many distinct values it takes for a single user. Anchor on your most stable identifier, such as `user_id`, and calculate the average number of distinct values of each other identifier per anchor value. A high average means the identifier is unstable for a user and will inflate the graph.

```sql
select avg(n_values) as avg_domain_userid_per_user

from (

  select user_id, count(distinct domain_userid) as n_values

  from atomic.events

  where user_id is not null

    and collector_tstamp > current_date - interval '30 days'

  group by user_id

) t;
```

Use the result to judge whether an identifier is stable enough to configure:

| Average distinct values per user | Interpretation                                       |
| -------------------------------- | ---------------------------------------------------- |
| Around one                       | Ideal: the identifier is stable for each user        |
| A small number, e.g. two to five | Acceptable: some churn across devices or sessions    |
| Tens or more                     | Reconsider: the identifier is unstable and high-cost |

## Reset identifiers on logout

When users share a device, a persistent identifier such as `domain_userid` can be associated with more than one user. To prevent this, reset the identifier when a user logs out.

For web browsers, call [`newSession`](/docs/sources/web-trackers/tracking-events/session/) or [`clearUserData`](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) on logout. This resets the `domain_userid` cookie so it isn't shared between users on the same browser. For more on how Identities handles shared devices, see [unique identifiers](/docs/identities/concepts/unique-identifiers/).

---

# Configure and manage Identities
> Configure identifiers, aliases, and event filters for Snowplow Identities identity resolution.
> Source: https://docs.snowplow.io/docs/identities/configuration/

Manage your Identities deployment and configure identifiers using [Snowplow Console](https://console.snowplowanalytics.com).

Navigate to **Identities** in the Console sidebar. If Identities is not yet deployed for your pipeline, you'll see a setup page. Click **Start configuration** to begin.

If Identities is already deployed, you'll see the Identities overview page with your current configuration. Click **Edit configuration** to make changes.

## Configure identifiers

Configure which fields from your events should be used as [identifiers](/docs/identities/concepts/#identifiers).

> **Tip:** The fields you choose affect both resolution quality and the cost and latency of running Identities. Before adding an identifier, read [choosing identifiers](/docs/identities/configuration/choosing-identifiers/) and validate its cardinality against your warehouse.

Each identifier type has:

| Field    | Description                                                                              | Required? |
| -------- | ---------------------------------------------------------------------------------------- | --------- |
| Name     | A unique name for this identifier type, e.g. `acme_user_id`                              | Yes       |
| Property | The event property to extract the identifier value from                                  | Yes       |
| Unique   | Whether this identifier should prevent merges between Snowplow IDs with different values | No        |
| Priority | The priority used when generating fallback IDs; higher is preferred                      | Yes       |

Only one identifier can be marked as unique.

A single event field can only be mapped to one identifier type. Identifier names must be unique.

### Add an identifier

Follow the steps in Console to create a new identifier.

> **Warning:** Removing an identifier affects identity resolution for all future events. Snowplow IDs that would have been connected through this identifier will appear as separate Snowplow IDs going forward. Historical identities won't change. This can't be undone.

### Enable cross-domain tracking aliases

If you use [cross-domain tracking](/docs/events/cross-navigation/), check the **Enable cross-domain tracking aliases** box. This automatically extracts `refr_domain_userid` as an identifier and maps it to `domain_userid` and `client_session_user_id`, so the same user is resolved across sites with different cookie domains.

> **Note:** For cross-domain tracking aliases to work, you must have identifiers named `domain_userid` or `client_session_user_id` (or both) defined in the identifiers section above.

## Set event filters

By default, Identities processes all events that flow through your pipeline. Use event filters to limit identity resolution to a subset of events.

Event filters are optional. Common reasons to filter events include:

- Processing only events from specific applications, using `app_id`
- Excluding server-side events that don't represent user activity
- Limiting to events from specific platforms

You can add multiple filters and combine them with **AND** (all conditions must match) or **ANY** (at least one condition must match) logic.

To add a filter, click **Add filter** and configure:

| Field    | Description                                        |
| -------- | -------------------------------------------------- |
| Property | The event property to filter on                    |
| Operator | `is one of` (include) or `is not one of` (exclude) |
| Value    | A list of values to match                          |

## Supported scale

Identities is self-serviceable from Console. The default deployment is sized for typical production event volumes and reasonable identifier configurations, where each configured identifier is stable for a user. Following the guidance in [choosing identifiers](/docs/identities/configuration/choosing-identifiers/) keeps your deployment within these defaults.

> **Warning:** High-volume pipelines aren't covered by the self-serve defaults and shouldn't be deployed without guidance. If you expect a large event volume, or your warehouse checks show high-cardinality identifiers you can't avoid, [contact Snowplow Support](https://support.snowplow.io/) before deploying Identities.

---

# Snowplow Identities dbt package
> The Snowplow Identities dbt package provides incremental models for identity resolution, mapping, and audit.
> Source: https://docs.snowplow.io/docs/identities/data-models/

Identities includes a [dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-identities-data-model/) to help you get value from your [Identities data](/docs/identities/concepts/#data-types). It transforms the raw identity entities and merge events in your warehouse into a set of derived tables for identity resolution, identifier lookup, and audit.

The Identities dbt package is currently standalone from our other dbt packages.

Check out the [Identities dbt package overview](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-identities-data-model/) for more information.

---

# Introduction to Snowplow Identities
> Snowplow Identities provides real-time identity resolution, stitching user identifiers together to create unified Snowplow IDs.
> Source: https://docs.snowplow.io/docs/identities/

Snowplow Identities provides real-time identity resolution. It stitches together user identifiers to create a unified view of each user, and adds a unified `snowplow_id` to each event.

Identities allows you to:

- Attribute anonymous events to authenticated users, both within the same session and across sessions and devices
- Connect behavior across web, iOS, and Android apps, when users authenticate with the same credentials on different platforms
- Build on [cross-navigation tracking](/docs/events/cross-navigation/) to identify users across domains
- Distinguish between separate users sharing the same device

Use the provided Identities data models to build identifier mapping tables. These tables help with key use cases such as marketing attribution, conversion funnel analysis, multi-touchpoint reporting, audience targeting, feature engineering, and personalization.

> **Note:** Identities is available on Cloud, and on AWS and GCP via [Private Managed Cloud](/docs/get-started/). Azure isn't supported.

## How Identities fits into the Snowplow pipeline

Identity resolution happens in real time as part of the [event enrichment process](/docs/fundamentals/).

![Diagram showing where Identities fits into the Snowplow pipeline](/assets/images/identities-pipeline-c9e6367cc110e4e7608f60c3f58641e3.png)

After all other configured enrichments run, the pipeline sends user identifiers in the event payload to the Identities service. It either links the identifiers to an existing Snowplow ID, or creates a new one. Enrich then adds the resolved Snowplow ID to the event in an [identity entity](/docs/identities/concepts/#identity-entity).

Some incoming identifiers will reveal that two previously separate Snowplow IDs actually belong to the same user. Identities will merge them in its graph database, and emit a [merge event](/docs/identities/concepts/#merge-events) directly into your enriched event stream.

Your Identities infrastructure is deployed into the same cloud as your pipeline. The core components are:

- **Identities API**: used for identity operations
- **Managed Postgres database**: persists the identity graph and performs graph operations. Snowplow uses [Aurora](https://aws.amazon.com/rds/aurora/) on AWS and [AlloyDB](https://cloud.google.com/products/alloydb) on GCP.

## Get started with Identities

Identities is self-serviceable from [Console](/docs/identities/configuration/) within its [supported scale](/docs/identities/configuration/#supported-scale).

To use Identities, start by deciding which user identifiers are most relevant to your use cases. Identities supports standard Snowplow identifiers such as `user_id` and `domain_userid`, as well as custom identifiers derived from any field in the event payload. Because your choice of identifiers affects both resolution quality and cost, follow the guidance in [choosing identifiers](/docs/identities/configuration/choosing-identifiers/) before you configure them.

After configuring Identities, set up the [Identities dbt package](/docs/identities/data-models/) to create your identity tables.

---

# Snowplow Documentation
> Snowplow CDI delivers validated, enriched event-level data to your warehouse, lake, or stream in real time. Define custom events with flexible schemas and distinguish AI agent traffic from human visitors.
> Source: https://docs.snowplow.io/docs/

**Deliver validated, enriched event data to your warehouse, lake, or stream in real time with Snowplow**.

[Get started with Snowplow CDI](/docs/get-started/)

[Your data foundation, deployed in your cloud or ours](/docs/get-started/)

[Snowplow Signals](/docs/signals/introduction/)

[Real-time customer context for AI agents and personalized experiences](/docs/signals/introduction/)

## Explore Snowplow Customer Data Infrastructure

Fundamentals

Learn about core Snowplow concepts

[Events](/docs/fundamentals/events/) [Entities](/docs/fundamentals/entities/) [Schemas](/docs/fundamentals/schemas/) [Atomic event fields reference](/docs/fundamentals/canonical-event/)

[Learn more →](/docs/fundamentals/)

Event collection

Manage your tracking

[Tracking plans](/docs/event-studio/tracking-plans/) [Source applications](/docs/event-studio/source-applications/) [Create a data structure](/docs/event-studio/data-structures/) [Snowtype](/docs/event-studio/implement-tracking/)

[Learn more →](/docs/event-studio/)

Destinations

Send data to warehouses, lakes, and third-party tools

[Warehouse and lake destinations](/docs/destinations/warehouses-lakes/) [Event forwarding](/docs/destinations/forwarding-events/) [How to query Snowplow data](/docs/destinations/warehouses-lakes/querying-data/) [Google Tag Manager Server-Side](/docs/destinations/forwarding-events/google-tag-manager-server-side/)

[Learn more →](/docs/destinations/)

Trackers

Instrument 20+ SDKs for web, mobile, and servers

[Get started on web](/docs/sources/web-trackers/tracker-setup/) [Track page views on web](/docs/sources/web-trackers/tracking-events/page-views/) [Get started on mobile](/docs/sources/mobile-trackers/installation-and-set-up/) [Track screen views on mobile](/docs/sources/mobile-trackers/tracking-events/screen-tracking/)

[Learn more →](/docs/sources/)

Pipeline

Configure your Collector and enrichments

[Collector](/docs/pipeline/collector/) [Enrichments](/docs/pipeline/enrichments/) [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)

[Learn more →](/docs/pipeline/)

Data modeling

Transform raw events into analytics-ready tables

[Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) [Attribution dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) [Incremental processing logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) [Identity stitching](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/)

[Learn more →](/docs/modeling-your-data/)

## Build with Snowplow

> **Tip:** Visit our [Developer Hub](https://snowplow.io/developer-hub/) for demo videos, blog posts, workshops, and more.

[Implement an abandoned browse system with composable CDP](/tutorials/abandoned-browse-ccdp/introduction/)

[Engage with customers that show intent to purchase but leave without buying](/tutorials/abandoned-browse-ccdp/introduction/)

[Set up the Unified Digital dbt package](/tutorials/unified-digital/intro/)

[Transform your raw web and mobile event data into derived tables](/tutorials/unified-digital/intro/)

[Use the Snowplow CLI MCP tool](/tutorials/snowplow-cli-mcp/introduction/)

[Let Claude or other MCP clients help with your data structure management](/tutorials/snowplow-cli-mcp/introduction/)

[See all tutorials ->](/tutorials/)

---

# Snowplow Community License FAQ
> Frequently asked questions about the Snowplow Community License terms, usage rights, and restrictions.
> Source: https://docs.snowplow.io/docs/licensing/community-license-faq/

> **Info:** This FAQ is not a substitute for reading [the license text](/community-license-1.0/).

This page explains some of the most common questions about the Snowplow Community License.

## What is the Snowplow Community License?

The Snowplow Community License is a source-available license based on the Confluent Community License, with changes related to rights and restrictions.

## How does the Snowplow Community License work in practice?

Under the Snowplow Community License, you can access the source code and modify it; but you cannot distribute it or use it to make a competing SaaS or on-premises offering. In the license, the things that you are not licensed to do are defined as an Excluded Purpose.

**The exact language**

For purposes of this Agreement, “Excluded Purpose” means making available any on-premises or distributed software product, software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or another similar online service, that competes with any Snowplow products or services that Snowplow or any of its affiliates provide using the Software.

For example, this does not allow the hosting of any Community License version of Snowplow’s Snowbridge product, or other software licensed under the Snowplow Community License, as part of an online service or distribution of on-premises offerings, that compete with Snowplow products or services.

If you are not doing what is excluded, this license change will not affect your current use.

## Is Snowplow moving away from open source?

No, we do not plan to wholly move away from open source, but like many companies today, we need to adjust our mix of open source and source-available offerings, to ensure we can continue to have the revenue and resources to support our open source community users.

We remain committed to the transparent processing of data, with the source code available for users to inspect and run for themselves as part of our “glass box” philosophy. Our source code will continue to be made available under one of the following: the Apache 2.0 license, the Snowplow Community License, the Snowplow Limited Use License Agreement, or the Snowplow Personal and Academic License. (See the [licensing overview](/docs/licensing/).)

That means our software will remain transparent and available.

## Can my company or I provide support to others who are running software under the Snowplow Community License?

This license is not intended to stop you from providing professional services, such as development, installation, or support, to others who are running software under the Snowplow Community License. Clients can engage you to do this work for them as long as they themselves are complying with our licenses.

## Can my company or I embed software under the Snowplow Community License in the software I distribute?

Only the MIT and Apache 2 licensed software from Snowplow can be embedded and distributed. The Snowplow Community License does not grant you rights to distribute software licensed under it. You do not have the right to distribute components licensed under the Snowplow Community License as part of an on-premises deployed package, and you do not have the right to use them as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), or Infrastructure-as-a-Service (IaaS).

## I have commercially licensed software from Snowplow. Does this impact me?

No, if you have entered into a separate commercial licensing with Snowplow, for example, buying a Snowplow CDI commercial product, then the commercial license terms you have agreed to will continue to govern your use of the software.

## How can I contact Snowplow in case of doubts?

For any further questions on licensing of Snowplow software, please send us an email at <ip-portfolio@snowplow.io>.

---

# Snowplow Contributor License Agreement
> Understand the Snowplow Contributor License Agreement requirements for contributing code to Snowplow projects.
> Source: https://docs.snowplow.io/docs/licensing/contributor-license-agreement/

As the project sponsor, Snowplow Analytics Ltd would like to ensure the long-term viability of Snowplow and its community. The Contributor License Agreement helps ensure everyone can enjoy Snowplow with confidence that Snowplow is here to stay.

Specifically, our Contributor License Agreements (CLAs) grant the contributor and Snowplow Analytics Ltd joint copyright interest in contributed code. Further, it provides assurance from the contributor that contributions are original work that does not violate any third-party license agreement. The agreement between contributors and project is explicit, so Snowplow users can be confident in the legal status of the source code and their right to use it.

## Our CLAs

For all contributions to Snowplow (be they software, bug fixes, configuration changes, documentation, or any other materials), we ask that contributors complete and sign a Contributor License Agreement.

We have two different CLAs, depending on whether you are contributing to Snowplow in a personal or professional capacity:

1. [Individual Contributor License Agreement v1.0](https://docs.google.com/forms/d/1J1FNYq9538ndzzcBdlCbxPo1yFiOY4mwalhDTSl1pgg/viewform)
2. [Software Grant and Corporate Contributor License Agreement v1.0](https://docs.google.com/forms/d/1ZUzz7lQJhs7oZqbkBL1bp0r048hAi7uIN6aLWCyZWWs/viewform)

## For lawyers

Our individual and corporate CLAs are based on the Contributor License Agreements published by the [Apache Software Foundation](http://www.apache.org/), with modifications:

1. Individual Contributor License Agreement ("Agreement") V2.0 ([cached here](/assets/apache-cla-2.0.pdf), or see the [newer V2.2 here](https://apache.org/licenses/icla.pdf))
2. [Software Grant and Corporate Contributor License Agreement ("Agreement") v r190612](http://www.apache.org/licenses/cla-corporate.txt)

If you have questions about Snowplow's own CLAs, please contact us at <legal@snowplowanalytics.com>.

---

# Snowplow licenses
> Information about the different licenses used in Snowplow components and projects, including the Community License, Limited Use License, and Personal and Academic License.
> Source: https://docs.snowplow.io/docs/licensing/

This section provides information about the different licenses used in Snowplow components and projects.

## Source-available components

> **Tip:** Since some of the below licenses are not approved by [OSI](https://opensource.org/licenses/), we do not refer to them as _Open Source_ licenses. However, the source code is still available.

| [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0)                                  | [Community License](/community-license-1.0/)                                                            | [Limited Use License](/limited-use-license-1.1/)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | [Personal & Academic License](/personal-and-academic-license-1.0/)                                          |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| _Usage and distribution for any purpose_                                                  | _Usage in non-competing products, no distribution_ ([FAQ](/docs/licensing/community-license-faq/))      | _Usage in non-production environments only, no distribution_ ([FAQ](/docs/licensing/limited-use-license-faq/))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | _Usage for personal and academic purposes only_ ([FAQ](/docs/licensing/personal-and-academic-license-faq/)) |
| - All tracker SDKs¹
- All Google Tag Manager templates
- Legacy data models (web, mobile) | * Core libraries (schema-ddl, common-streams, etc)
* Developer tools (Chrome Inspector, `igluctl`, etc) | All pipeline apps:* [Collector](/docs/api-reference/stream-collector/) _(3.0.0+)_
* [Enrich](/docs/api-reference/enrichment-components/) _(4.0.0+)_
* [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) _(0.12.0+)_
* [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) _(6.0.0+)_
* [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)
* [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/) _(2.0.0+)_
* [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) _(0.2.0+)_
* Other loaders
* [Snowbridge](/docs/api-reference/snowbridge/) _(2.4.0+)_
* Community Edition terraform modules _(2024+)_ | All data models (except web and mobile)                                                                     |

¹As an exception, [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow-javascript-tracker) is distributed under [BSD 3 Clause](https://opensource.org/licenses/BSD-3-Clause). This tracker was originally based on Anthon Pang’s `piwik.js`, and has maintained the same license for distribution.

> **Tip:** When in doubt, consult each component’s GitHub repository for the LICENSE file.

## Proprietary components

[Snowplow CDI](https://snowplow.io/) is built upon the above components, but adds a vast set of proprietary, closed source ones (UI, API, highly available deployment logic, and so on). These are only available under a commercial license for Snowplow customers.

---

# Snowplow Limited Use License FAQ
> Frequently asked questions about the Snowplow Limited Use License Agreement terms and permitted uses.
> Source: https://docs.snowplow.io/docs/licensing/limited-use-license-faq/

> **Info:** This FAQ is not a substitute for reading the license text:
>
> - [version 1.0](/limited-use-license-1.0/)
> - [version 1.1](/limited-use-license-1.1/)

This page explains some of the most common questions about the Snowplow Limited Use License Agreement.

## Why was the Snowplow Limited Use License Agreement introduced?

Previously, many of our Snowplow core pipeline components and dbt models were licensed as open source under the Apache 2.0 license. For more than a decade, our team has invested significant time and resources into researching, developing, and maintaining this code in an open source manner.

Looking forward, we have an exciting technology roadmap that will deliver additional value to all of our core pipeline users. It includes plans to reduce the cost of running the pipeline at scale, improve processing speed, provide more flexibility for custom in-stream processing steps, and support non-HTTP protocols for more efficient data ingestion from server-side and mobile devices. In order to fund these developments, we are asking Snowplow users who are running the Snowplow behavioral data pipeline in production to pay for the value they receive in return.

## What is the Snowplow Limited Use License Agreement?

Snowplow is moving new versions of the Snowplow pipeline components to a source-available license. This license is based on the [Snowplow Community License](/docs/licensing/community-license-faq/), with additional deployment and provisioning restrictions. Notably, this license includes restrictions that prohibit deploying the Snowplow software for production workloads.

## How does the Snowplow Limited Use License Agreement work in practice?

Under the Snowplow Limited Use License Agreement, you can access the source code and modify it, but you cannot distribute it or use it to make a competing SaaS or on-premises offering. Nor can you deploy the Snowplow software in production. Non-production use of the software, such as with development and quality assurance pipelines, are allowed.

**Excerpt: the exact language in the license**

Licensee is not granted the right to, and Licensee shall not, exercise the License for any Competing Use, and Licensee may exercise the License only for Non-Production Use or Non-Commercial Use.

- “Competing Use” is making available any on-premises or distributed software product, or any software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or other similar online service, that competes with any products or services that Snowplow or any of its affiliates provides using the Software.

- “Non-Production Use” means any use of the Software to process test or synthetic data to evaluate the sufficiency of the Software for use by Licensee.

- “Non-Commercial Use” is only: (a) personal use for research, experiment, personal study, or hobby projects, without any anticipated commercial application, or (b) use for teaching purposes by lecturers of a school or university.

If you are not doing what is excluded, this license change will not affect your current use.

## Is Snowplow moving away from open source?

No, we do not plan to wholly move away from open source, but like many companies today, we need to adjust our mix of open source and source-available offerings, to ensure we can continue to have the revenue and resources to support our open source community users.

We remain committed to the transparent processing of data, with the source code available for users to inspect and run for themselves as part of our “glass box” philosophy. Our source code will continue to be made available under one of the following: the Apache 2.0 license, the Snowplow Community License, the Snowplow Limited Use License Agreement, or the Snowplow Personal and Academic License. (See the [licensing overview](/docs/licensing/).)

That means our software will remain transparent and available.

## When do these license changes take effect?

The Snowplow Limited Use License Agreement (SLULA) version 1.0 was rolled out in January, 2024. Version 1.1 is rolled out in December, 2024.

## What changed in version 1.1?

The Snowplow Limited Use License v1.1 (SLULA v1.1) simplifies the terms of acceptable use. It allows you to run SLULA-licensed software in test environments or for academic or research purposes. It does not allow you to run in production or commercial environments. To run in these environments you must obtain a commercial license from Snowplow.

This change does not take away your rights under SLULA 1.0, for versions that were released under that license.

### Summary of changes

Here is a summary of the specific changes in SLULA v1.1 over the v1.0 license:

- Section 1.1 was updated to clarify acceptable use as “Non-Production” or “Non-Commercial” use only
- Section 1.2 and its subsections were replaced completely. We have removed the “Highly Available” provision in the v1.0 license. If you are running the software in production in either a non-highly available or highly available manner, you must obtain a commercial license
- Section 1.2.2 now defines the definitions of “Non-Production Use”. You may use the software on test or synthetic data in a non-commercial manner
- Section 1.2.3, a new section, now defines “Non-Commercial Use”, which includes research, experimentation, personal study or hobby projects with no anticipated commercial application

## What components of Snowplow are offered under this new license?

See the [licensing overview](/docs/licensing/) page.

## What does this mean for me?

If you currently run Snowplow’s open source pipeline code in production (on live data) or in a competitive manner, and wish to use versions of Snowplow released under the SLULA, please contact the team [here](https://snowplow.io/snowplow-oss-license-change/).

If you are a current user of Snowplow open source software but do not run the core pipeline code in production or in a competitive manner, you may continue to use new Snowplow Limited Use License Agreement versions of Snowplow.

If you are a commercial Snowplow CDI Private Managed Cloud or CDI Cloud customer, this license change does not impact your use of Snowplow.

### Examples

Here are some examples what is or is not acceptable under the terms of the SLULA license version 1.1.

#### Allowed

- I want to download and deploy Snowplow internally in either single point of failure (SPOF) or highly available (HA) manner, and run synthetic data through it, for example for product evaluation or load testing purposes
- I want to download the Snowplow software and do research on it (university, hobby, etc), replaying old (real) data, or synthetic data through the pipeline
- I want to download and deploy in a Highly Available (HA) or Single Point of Failure (SPOF) manner Snowplow and publicly expose the pipeline to the internet, capturing events from my personal non-commercial website

#### Not allowed. Requires a commercial license

- I want to download and deploy in a HA or SPOF manner Snowplow and publicly expose the pipeline to the internet, capturing events from my personal website where I have a hobby business selling photos or art or crafts for my ‘side hustle’
- I want to download and deploy in a HA or SPOF manner Snowplow and publicly expose the pipeline to the internet, capturing events from my company website or mobile app.
- I want to download and deploy in a HA or SPOF manner Snowplow and publicly expose the pipeline to the internet, capturing events from my company website or mobile app to test if Snowplow can handle the event volume for my company on live data

## Can my company or I provide support to others who are running software under the Snowplow Limited Use License Agreement?

This license is not intended to stop you from providing professional services, such as development, installation, or support, to others who are running software under the Snowplow Limited Use License Agreement. Clients can engage you to do this work for them as long as they themselves are complying with our licenses. We do not consider this a competitive use.

## Can my company or I embed software under the Snowplow Limited Use License Agreement in the software I distribute?

Only the MIT/BSD and Apache 2 licensed software from Snowplow can be embedded and distributed. The Snowplow Limited Use License Agreement does not grant you rights to distribute software licensed under it. You do not have the right to distribute components licensed under the Snowplow Limited Use License Agreement as part of an on-premises deployed package, and you do not have the right to use them as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), or Infrastructure-as-a-Service (IaaS).

## How can I contact Snowplow in case of doubts?

For any further questions on licensing of Snowplow software, please send us an email at <ip-portfolio@snowplow.io>.

---

# Snowplow Personal and Academic License FAQ
> Frequently asked questions about the Snowplow Personal and Academic License eligibility and terms.
> Source: https://docs.snowplow.io/docs/licensing/personal-and-academic-license-faq/

> **Info:** This FAQ is not a substitute for reading [the license text](/personal-and-academic-license-1.0/).

This page explains some of the most common questions about the Snowplow Personal and Academic License.

## What is the SPAL?

The SPAL is a license from Snowplow Analytics, designed for products and components where we want to make the source available for non-commercial use. Some of our code is available under open source licenses.

## Which license is the SPAL based on?

The SPAL is based on the [PolyForm Noncommercial License](https://polyformproject.org/licenses/noncommercial/1.0.0).

## I have commercially licensed software from Snowplow. Does this impact me?

No. If you have entered into a separate commercial licensing with Snowplow, for example, buying a Snowplow CDI commercial product, then the commercial license terms you have agreed to will continue to govern your use of the software.

## Why make the source available?

First, at Snowplow, we have a design principle that all data processing must be ‘glass box’: it must be possible as a data practitioner running Snowplow to understand exactly how the data processing affects output from end to end.

Second, we want to encourage the use of this code as a teaching aid for students.

Third, we want to encourage data practitioners to try out this code for personal uses without needing a commercial relationship with us.

## I am a student or just tinkering, can I run this code?

Yes, the SPAL encourages usage of the code by educators and students for educational purposes. If you are a data practitioner seeking to use this personally, for example on a blog or hobby project, that is also fine.

## I am an educational institution wanting to use this in the running of my business, can I?

No. That is a commercial use. Please contact us for commercial licensing, see [below](#i-want-to-use-this-code-commercially-what-is-the-next-step).

## Can my company or I provide support to others who are running software under the SPAL?

This license is not intended to stop you from providing professional services, such as development, installation, or support, to others who are running software under the SPAL. Clients can engage you to do this work for them as long as they themselves are complying with our licenses.

## Why don’t you release the code under an open source license?

We need to run a sustainable business. Snowplow’s open source software development is funded by our commercial efforts, which include charging to use code released under the SPAL for commercial uses.

## Is Snowplow moving away from open source?

No, we do not plan to wholly move away from open source, but like many companies today, we need to adjust our mix of open source and source-available offerings, to ensure we can continue to have the revenue and resources to support our open source community users.

We remain committed to the transparent processing of data, with the source code available for users to inspect and run for themselves as part of our “glass box” philosophy. Our source code will continue to be made available under one of the following: the Apache 2.0 license, the Snowplow Community License, the Snowplow Limited Use License Agreement, or the Snowplow Personal and Academic License. (See the [licensing overview](/docs/licensing/).)

That means our software will remain transparent and available.

## Can I contribute to this software?

Yes, we welcome community contributions. However, you will need to agree to our [Contribution License Agreement terms](/docs/licensing/contributor-license-agreement/).

## As a commercial user, can I make edits to code?

The SPAL does not allow commercial use. However, if you want to use the code for commercial purposes, we can discuss a license that will allow you to use and modify the code for your commercial purposes.

## I want to use this code commercially, what is the next step?

Please email us at <sales@snowplow.io>.

## Where can I ask my other IP-related questions not covered here?

Please email us at <ip-portfolio@snowplow.io>.

---

# CLI MCP server
> Use the Snowplow CLI MCP server to interact with your tracking plans through AI assistants.
> Source: https://docs.snowplow.io/docs/llms-support/cli-mcp-server/

The [Snowplow CLI](/docs/api-reference/snowplow-cli/) includes a local Model Context Protocol (MCP) server that enables natural language interaction with AI assistants for creating, validating, and managing your Snowplow tracking plans. This allows you to:

- Create and validate data structures through conversation
- Analyze tracking requirements and suggest implementations
- Validate tracking plans and source applications

> **Info:** For a step-by-step guide to using the Snowplow CLI MCP server, see the [MCP tutorial](/tutorials/snowplow-cli-mcp/introduction/).

The MCP server allows AI assistants to interact with the Snowplow CLI using the `snowplow-cli mcp` command.

## Install the MCP server

The MCP server is included with [Snowplow CLI](/docs/api-reference/snowplow-cli/). Start by installing the CLI and authenticating to your Snowplow account.

To connect your AI client to the MCP server, provide a configuration object. The exact structure depends which client you are using, and whether you're connecting to the Snowplow CLI installed locally or via npx. For example:

**Claude Desktop:**

Add the following to your Claude Desktop configuration file, found at:

- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Windows: `%APPDATA%\Claude\claude_desktop_config.json`

For direct Snowplow CLI connection:

```json
{

  "mcpServers": {

    "snowplow-cli": {

      "command": "snowplow-cli",

      "args": ["mcp"]

    }

  }

}
```

Using npx:

```json
{

  "mcpServers": {

    "snowplow-cli": {

      "command": "npx",

      "args": ["-y", "@snowplow/snowplow-cli", "mcp"]

    }

  }

}
```

> **Note:** Claude Desktop requires additional filesystem access to create and modify files.

**VS Code:**

Add to `.vscode/mcp.json` in your workspace.

For direct Snowplow CLI connection:

```json
{

  "servers": {

    "snowplow-cli": {

      "type": "stdio",

      "command": "snowplow-cli",

      "args": ["mcp"]

    }

  }

}
```

Using npx:

```json
{

  "servers": {

    "snowplow-cli": {

      "type": "stdio",

      "command": "npx",

      "args": ["-y", "@snowplow/snowplow-cli", "mcp"]

    }

  }

}
```

**Cursor:**

Add to `.cursor/mcp.json` in your workspace:

```json
{

  "mcpServers": {

    "snowplow-cli": {

      "command": "snowplow-cli",

      "args": ["mcp", "--base-directory", "."]

    }

  }

}
```

***

After adding the configuration, restart the client. The Snowplow CLI MCP tools should be available for use.

## Provided tools

The Snowplow CLI MCP server provides these tools to your AI assistant:

- `get_context`: to call at the start of each conversation, this tool retrieves the underlying Snowplow rules that define how Snowplow components should be structured
- `get_uuid`: generates valid v4 UUIDs that are required by many Snowplow components
- `validate_data_structures`: the assistant will use this validation tool after creating or modifying any data structure.
- `validate_data_products`: the assistant will use this validation tool after creating or modifying any tracking plan or source application file

The `get_context` tool is the only one you might need to call manually.

## Using the tools

To get AI support with your Snowplow tracking plans, start a new conversation and ask the assistant to call `get_context`. It might call this tool automatically if it realises it needs the Snowplow knowledge.

You can then ask it for help. It has access to your existing data structures, tracking plans, or source application definitions. The assistant will automatically validate any file changes using the included validation tools.

Use the standard [Snowplow CLI commands](/docs/api-reference/snowplow-cli/reference/) to publish your changes to [Console](https://console.snowplowanalytics.com) when ready, using the standard Snowplow CLI commands such as `snowplow-cli ds publish`.

> **Note:** The MCP server creates and validates files on your local filesystem only.

---

# Snowplow Assistant
> The Snowplow Assistant is an AI assistant built into Snowplow BDP Console that lets you manage your tracking plans, pipelines, and data quality through natural language.
> Source: https://docs.snowplow.io/docs/llms-support/console-agent/

The Snowplow Assistant is an AI assistant embedded in [Snowplow Console](https://console.snowplowanalytics.com). It lets you manage your tracking implementation, monitor pipelines, troubleshoot issues, and configure Signals through natural language conversation.

## How it works

The Snowplow Assistant runs as a chat interface inside the Console. When you send a message, the assistant determines which tools and skills to use, calls the relevant Snowplow APIs on your behalf, and returns the results in a conversational format.

### Permissions

The Snowplow Assistant operates with **your existing Console permissions**. It authenticates using your current session and can only perform actions that your account is authorized to do. It cannot escalate privileges or access resources outside your organization.

Any action that modifies data (creating a schema, updating an enrichment, deleting an alert) requires your explicit confirmation before the assistant proceeds.

### Data privacy

The Snowplow Assistant does not have access to personal data or PII such as names or email addresses. It operates only on Snowplow configuration and metadata (schemas, tracking plans, pipeline settings, failed event error details, and similar).

> **Warning:** To provide this functionality, the content of your messages and the configuration data the assistant retrieves on your behalf are sent to a third-party LLM provider for processing. Do not enter personal data, credentials, or other sensitive information into the chat.

### Enabling and disabling the assistant

You can enable or disable the Snowplow Assistant for your organization from the **Settings** section of the Snowplow Console. When disabled, the chat interface does not appear in the Console for any user in your organization.

> **Note:** Only users with the Administrator role can accept the terms and conditions and enable or disable the Snowplow Assistant.

When enabling the feature for the first time, an admin user must accept the terms and conditions related to LLM usage within the Snowplow product.

### Usage Limits

The Snowplow Assistant has weekly refreshing token limits at the organisation level. The limit is set at a level that under normal usage you will not reach it.

## Capabilities

The assistant provides support across several areas of the Console. It loads specialized skills on demand based on your request.

### Tracking design

Design and implement your event tracking using natural language:

- Create and manage [data structures](/docs/fundamentals/schemas/) (Iglu schemas)
- Create and manage [event specifications](/docs/event-studio/tracking-plans/event-specifications/) with entity contexts
- Create and manage [tracking plans](/docs/event-studio/tracking-plans/)
- Search [Iglu Central](https://github.com/snowplow/iglu-central) for reusable public schemas
- Browse the data catalog to understand what is already being tracked

### Implementation guidance

Get help instrumenting Snowplow trackers in your applications:

- Generate copy-paste-ready tracking code for web, mobile, and server-side platforms
- Look up schema definitions and event specifications for correct implementation
- Follow a guided workflow from instrumentation through to production deployment

### Pipeline management

Explore and understand your pipeline infrastructure:

- List pipelines with their status, collector endpoints, and labels
- View real-time pipeline metrics including throughput, enrichment latency, and bad row rates
- Review collector configuration (CNAME, cookie policy, DNS)
- List and manage enrichments
- View mini and Micro development instances

### Troubleshooting

Diagnose and resolve pipeline issues:

- Investigate [failed events](/docs/fundamentals/failed-events/) with detailed error breakdowns
- Analyze schema validation failures and enrichment errors
- Drill into specific errors by app ID and tracker version
- Get targeted recommendations for fixes

### Data quality

Manage alerts for data quality issues:

- Create and manage data quality alerts via email or Slack
- Filter alerts by app ID, issue type, or data structure
- Update or remove existing alert configurations

### Console operations

Perform day-to-day management tasks:

- View and manage source applications, including their platform, app IDs, and entity definitions
- Navigate directly to relevant Console pages from the chat
- Search and browse the Snowplow documentation

### Signals

If [Signals](/docs/signals/concepts/) is enabled for your organization, the assistant can help you configure and manage it:

- Define and manage attribute keys and attribute groups
- Create services for pull-based attribute access
- Set up interventions for push-based actions
- Test attribute groups against sample data
- Publish and unpublish configurations
- Query real-time attributes for specific users

---

# Working with AI
> How Snowplow supports LLM and AI-powered workflows, from real-time behavioral context to tracking plan management and AI-readable documentation.
> Source: https://docs.snowplow.io/docs/llms-support/

Snowplow supports agentic and LLM-powered workflows in several ways.

## Snowplow Assistant

The [Snowplow Assistant](/docs/llms-support/console-agent/) is an AI assistant embedded in Snowplow Console. It lets you manage your tracking implementation, monitor pipelines, troubleshoot issues, and configure Signals through natural language conversation, using your existing Console permissions.

## Snowplow MCP server

The [Snowplow MCP server](/docs/llms-support/snowplow-mcp/) is a remote MCP server that connects AI assistants to your Snowplow Console account. It provides broad access to Console functionality, including pipelines, failed events, event specifications, the data catalog, and more.

## CLI MCP server

The [Snowplow CLI MCP server](/docs/llms-support/cli-mcp-server/) runs locally and connects AI assistants to your tracking plan files on disk. It is included in the [Snowplow CLI](/docs/api-reference/snowplow-cli/) and is focused on tracking plan design and validation.

## Documentation index in `llms.txt`

This documentation follows the [`llms.txt` standard](https://llmstxt.org/), providing structured information to help an LLM use the site.

An index is available at [`llms.txt`](/llms.txt).

An extended version is also available at [`llms-full.txt`](/llms-full.txt), that includes the complete text of the current pages. For token efficiency, sections relating to older versions of components aren't included in this file. These sections are still listed in the `llms.txt` index, labeled as `[previous version]`.

The `llms-full.txt` file is very large. It might be more effective to access individual pages as needed, using the Markdown access method described below.

## Documentation pages as Markdown

Every documentation page is available as Markdown. To download a page's content, use the **Download** or **Copy Markdown** buttons above the page title.

Following the `llms.txt` standard, you can access the Markdown page directly by changing the trailing `/` in the URL to `.md`. For example:

- HTML: `https://docs.snowplow.io/docs/signals/concepts/`
- Markdown: `https://docs.snowplow.io/docs/signals/concepts.md`

Let your LLM know that this format is available, so it can retrieve content efficiently.

## Signals

Use [Signals](/docs/signals/concepts/) to provide real-time behavioral context to your AI applications. It computes user attributes from your event stream and warehouse data, and makes them available to your applications via the [Profiles Store](/docs/signals/concepts/#profiles-store) API.

---

# Snowplow MCP server
> Connect your AI assistant to the Snowplow MCP server for natural language console management.
> Source: https://docs.snowplow.io/docs/llms-support/snowplow-mcp/

Snowplow MCP is a remote [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that lets AI assistants interact with your Snowplow Console account using natural language.

The MCP server exposes the same set of tools as the Snowplow Assistant, but in your choice of harness and model. The server authenticates through your existing Snowplow Console login via OAuth, or with a Console API key to avoid re-authenticating — see [Authentication](#authentication).

## One-line install (Claude Code, Cursor)

The recommended way to install Snowplow MCP is with the [`plugins` CLI](https://github.com/vercel-labs/plugins), a vendor-neutral installer that auto-detects supported AI tools. From any terminal, run:

```bash
npx plugins add snowplow/skills
```

This installs the Snowplow MCP server plus six bundled skills — `tracking-design`, `implementation-guidance`, `signals`, `pipeline-infrastructure`, `console-operations`, and `troubleshooting` — into any detected agent tool in a single step. The plugin source lives at [github.com/snowplow/skills](https://github.com/snowplow/skills).

The `plugins` CLI auto-detects [Claude Code](https://claude.com/product/claude-code) and [Cursor](https://cursor.com/). Pass `--target claude-code` or `--target cursor` to scope the install to a specific tool.

On first use, your tool opens a browser for OAuth against [Snowplow Console](https://console.snowplowanalytics.com) — see [Authentication](#authentication) for the permissions model.

> **Tip:** The bundled skills are loaded on demand by the model. Claude or Cursor automatically engages the relevant skill when you ask about tracking design, pipeline troubleshooting, or any of the other areas. There is no slash command to run.

## Install with the Claude Code plugin marketplace

If you use [Claude Code](https://claude.com/product/claude-code), you can install the Snowplow plugin from its native marketplace instead. Register the marketplace and install the plugin from inside Claude Code:

```text
/plugin marketplace add snowplow/skills

/plugin install snowplow@snowplow
```

This installs the Snowplow MCP server plus the same six bundled skills as the [`plugins` CLI](#one-line-install-claude-code-cursor). Claude Code reads the plugin directly from the [repository](https://github.com/snowplow/skills) and updates it automatically when the repository changes.

Prefer a single command across Claude Code, Cursor, and Codex? Use the vendor-neutral [`plugins` CLI](#one-line-install-claude-code-cursor) instead.

## Configure the MCP server

Configure Snowplow MCP manually using the snippets below for your tool of choice.

**Claude.ai:**

Claude.ai supports remote MCP servers as custom connectors. To add Snowplow MCP:

1. Open [Claude.ai](https://claude.ai) and navigate to **Settings**.

2. Under **Connectors**, click **Add connector**.

3. Fill in the connector details:

   - **Name**: `Snowplow MCP`
   - **URL**: `https://console.snowplowanalytics.com/api/agent/mcp`

4. Expand **Advanced settings** and enter the OAuth Client ID: `NxCcdyu13Cr4umnIYw70evvUyRXRvyWf`

5. Click **Add**.

![Connector details dialog showing the Snowplow MCP name and URL with OAuth Client ID in advanced settings](/assets/images/claude-connector-details-66d9031a5fa7eea656448656a6fe97f7.png)

Claude.ai will redirect you to the Snowplow login page to authorize access.

**Claude Desktop:**

The custom connector method also works for Claude Desktop. But in case you don't have access to custom connectors, you can configure it directly in Claude Desktop using [`mcp-remote`](https://www.npmjs.com/package/mcp-remote)

Add the following to your Claude Desktop configuration file:

- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Windows: `%APPDATA%\Claude\claude_desktop_config.json`

```json
{

  "mcpServers": {

    "snowplow-mcp": {

      "command": "npx",

      "args": [

        "-y",

        "mcp-remote",

        "https://console.snowplowanalytics.com/api/agent/mcp",

        "3334",

        "--static-oauth-client-info",

        "{\"client_id\":\"NxCcdyu13Cr4umnIYw70evvUyRXRvyWf\"}"

      ]

    }

  }

}
```

Restart Claude Desktop after saving. On startup Claude Desktop opens a browser window for Snowplow login and caches the token locally for subsequent sessions.

**Claude Code:**

Add a remote MCP server to Claude Code by running:

```bash
claude mcp add snowplow-mcp \

  --transport http \

  --client-id NxCcdyu13Cr4umnIYw70evvUyRXRvyWf \

  --callback-port 3334 \

  https://console.snowplowanalytics.com/api/agent/mcp
```

Run the "mcp" command, select the "snowplow-mcp" and then "Authenticate". This should open a browser page for Snowplow login.

**Codex CLI:**

Add a remote MCP server to Codex CLI by running:

```bash
codex mcp add snowplow-mcp -- \

  npx mcp-remote \

  https://console.snowplowanalytics.com/api/agent/mcp \

  3334 \

  --static-oauth-client-info \

  '{"client_id":"NxCcdyu13Cr4umnIYw70evvUyRXRvyWf"}'
```

Codex will redirect you to the Snowplow login page to authorize access.

**Codex UI:**

1. Open Codex and navigate to **Settings** > **MCP servers** > **Add server** > **STDIO**.

2. Fill in the server details:

   - **Name**: `Snowplow MCP`

   - **Command to launch**: `npx`

   - **Arguments** (one per row):

     1. `mcp-remote`
     2. `https://console.snowplowanalytics.com/api/agent/mcp`
     3. `3334`
     4. `--static-oauth-client-info`
     5. `{"client_id":"NxCcdyu13Cr4umnIYw70evvUyRXRvyWf"}`

3. Click **Save**.

![Connector details dialog showing the Snowplow MCP configuration in Codex](/assets/images/codex-ui-48c45dec92c95c8d1f90f3ca4908addd.png)

Codex will redirect you to the Snowplow login page to authorize access.

**Cursor:**

1. Open Cursor and navigate to **Settings** > **Cursor Settings** > **Tools & MCPs**.
2. Click **New MCP Server**. This opens the `mcp.json` configuration file.
3. Add the following configuration:

```json
{

  "mcpServers": {

    "snowplow-mcp": {

      "command": "npx",

      "args": [

        "-y",

        "mcp-remote",

        "https://console.snowplowanalytics.com/api/agent/mcp",

        "3334",

        "--static-oauth-client-info",

        "{\"client_id\":\"NxCcdyu13Cr4umnIYw70evvUyRXRvyWf\"}"

      ]

    }

  }

}
```

4. Save the file and return to **Tools & MCPs**. The server should appear in the list.

Cursor will redirect you to the Snowplow login page to authorize access.

***

## Authentication

OAuth and API keys are the two ways to authenticate with your Snowplow Console account. Both give the assistant access to one organization at a time, limited to what the credentials allow.

### OAuth

The configurations above use OAuth. On first connection, your tool opens a browser to authorize against [Snowplow Console](https://console.snowplowanalytics.com), and the assistant then operates with the same permissions as your user account — it can access and modify only what you can.

OAuth uses the organization your account belongs to by default. To target a different organization that your user has access to, add an `X-Org-Id` header with the organization ID, found on the **Manage organization** page in [Console](https://console.snowplowanalytics.com/settings). With `mcp-remote`, append these entries to the `args` array of any configuration above:

```json
"--header",

"X-Org-Id:<ORGANIZATION_ID>"
```

### API key

Authenticate with a Snowplow Console API key to avoid repeated browser logins. Unlike OAuth tokens, which expire and prompt a fresh login, an API key is a long-lived credential passed as HTTP headers, so the connection keeps working without interruption.

First, [create an API key](/docs/account-management/#create-an-api-key) in Console and note the API key ID and the API key. From the same **Manage organization** page, copy your organization ID. Then configure the server with the `X-Org-Id`, `X-Api-Key-Id`, and `X-Api-Key` headers:

```json
{

  "mcpServers": {

    "snowplow-mcp": {

      "command": "npx",

      "args": [

        "-y",

        "mcp-remote",

        "https://console.snowplowanalytics.com/api/agent/mcp",

        "--header",

        "X-Org-Id:<ORGANIZATION_ID>",

        "--header",

        "X-Api-Key-Id:<API_KEY_ID>",

        "--header",

        "X-Api-Key:<API_KEY>"

      ]

    }

  }

}
```

> **Warning:** By default, API keys are created with all permissions, which may be broader than you intend for an assistant. The assistant can perform any action the key allows. If you need the assistant to operate with limited permissions, use [OAuth](#oauth) instead, which is scoped to your user account. Store the key securely and only use this method in environments you trust.

## Capabilities of the server

The MCP server gives your AI assistant read and write access to the main areas of Snowplow Console. The full list of tools is visible in your MCP client after connecting.

- **Data structures and schemas** — browse, inspect, and create [data structures](/docs/fundamentals/schemas/), and search [Iglu Central](https://github.com/snowplow/iglu-central) for reusable public schemas.
- **Event specifications and tracking plans** — manage [event specifications](/docs/event-studio/tracking-plans/event-specifications/) and [tracking plans](/docs/event-studio/tracking-plans/), and view event volume metrics.
- **Source applications** — manage [source applications](/docs/event-studio/source-applications/) and their associated entities.
- **Pipelines** — inspect [pipeline](/docs/pipeline/) configuration, health metrics, and collector settings, including [Snowplow Micro](/docs/testing/snowplow-micro/) instances.
- **Failed events and data quality** — investigate [failed events](/docs/fundamentals/failed-events/) and manage data quality alerts.
- **Enrichments** — view and update [enrichment](/docs/pipeline/enrichments/) configurations.
- **Data catalog** — browse and search tracked data structures.
- **Signals** — manage the full [Signals](/docs/signals/) workflow, including attribute groups, services, interventions, and publishing to compute engines.
- **Documentation** — fetch pages from the Snowplow documentation site for quick reference.

## Example prompts

After connecting, try asking your assistant:

- "What pipelines do I have and what's their current status?"
- "Show me the failed events from the last 24 hours"
- "Create a new event specification for a signup\_completed event"
- "What enrichments are enabled on my production pipeline?"
- "Search the data catalog for anything related to ecommerce"

---

# Migrate from Adobe Analytics to Snowplow
> Migrate from Adobe Analytics to Snowplow with guidance on event tracking, eVars, props, and warehouse modeling.
> Source: https://docs.snowplow.io/docs/migration-guides/adobe-analytics/

This guide is to help technical implementers migrate from Adobe Analytics to Snowplow. For more advice on tracking plans, check out our [best practices guide](/docs/fundamentals/tracking-design-best-practice/).

## Platform differences

There are significant differences between Adobe Analytics and Snowplow as data platforms. For migration, it's important to understand how Snowplow structures events differently from Adobe Analytics. This affects how you'll implement tracking and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Adobe Analytics                                                                                       | Snowplow                                                                                                                                                              |
| ----------------------- | ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capturing user behavior | Uses numbered variables                                                                               | Direct tracker calls for built-in or custom [event](/docs/fundamentals/events/) types, including contextual data as reusable [entities](/docs/fundamentals/entities/) |
| Warehouse tables        | Data Feeds export to flat files with numbered columns (prop1-prop75, eVar1-eVar250, event1-event1000) | In [most warehouses](/docs/destinations/warehouses-lakes/), one big table with columns for every event or entity; in Redshift, one table per custom event or entity   |
| Data validation         | Processing rules and VISTA rules for transformation; no schema validation                             | All events and entities defined by JSON schemas                                                                                                                       |

### Event structure

Adobe Analytics uses two primary [tracking methods](https://experienceleague.adobe.com/en/docs/analytics/implementation/vars/functions/overview#): `s.t()` for page views and `s.tl()` for custom link tracking. Data is captured through props, eVars, and events.

Each of these types uses numbered variables with different persistence scopes:

- Props (`s.prop1-75`): hit-scoped traffic variables that expire after the hit
- eVars (`s.eVar1-250`): conversion variables with configurable expiration (hit, visit, visitor, or specific events)
- Events (`event1-1000`): counters and currency values for success metrics

This numbered variable system requires external documentation to map variable numbers to business meanings. For example, `eVar5` might represent "campaign code" in one implementation and "product category" in another.

Here's an example Adobe Analytics web ecommerce implementation. The code sets various variables before calling the tracking method:

```javascript
// Adobe Analytics purchase tracking

s.pageName = 'Order Confirmation';

s.events = 'purchase,event1';

s.products = ';SKU_123;1;25.42';

s.eVar1 = '12345';           // transaction_id

s.eVar2 = 'USD';             // currency

s.prop1 = 'Electronics';     // category

s.purchaseID = '12345';

s.t();
```

The `s.products` variable uses a specific syntax with semicolon delimiters: `category;product;quantity;price`. Success events like `purchase` are tracked in `s.events`. Context is spread across numbered props and eVars, requiring documentation to understand what each variable represents.

To track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event:

```javascript
snowplow('trackTransaction', {

  transaction_id: 'T_12345',

  revenue: 25.42,

  currency: 'USD',

  products: [{

    id: 'SKU_123',

    name: 'Product Name',

    category: 'Electronics',

    price: 25.42,

    quantity: 1

  }]

});
```

As well as the tracking code looking different, there are key differences in the warehouse loading and data modeling:

- Snowplow processes this as a `transaction` [event](/docs/fundamentals/events/) with one `product` [entity](/docs/fundamentals/entities/)

- The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser

- In BigQuery, Databricks, or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for the `transaction` event and a column for each entity

  - Columns have type `OBJECT` in Snowflake, and `REPEATED RECORD` in BigQuery
  - In Redshift, each event and entity is loaded into its own table

Snowplow tracker SDKs provide built-in methods for tracking page views and screen views, along with many other kinds of events. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling and common analytics use cases.

### Tracking comparison

This table shows how Adobe Analytics tracking maps to Snowplow tracking:

| Adobe Analytics Method | Adobe Example                                                    | Snowplow Implementation                                                                                                                                                                 |
| ---------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `s.t()` (page view)    | `s.pageName='Home'; s.t();`                                      | Use `trackPageView`. The tracker automatically captures page title, URL, and referrer.                                                                                                  |
| `s.tl()` (link click)  | `s.tl(this,'o','Button Click');`                                 | Use `trackLinkClick` for links, or define a custom schema and track with `trackSelfDescribingEvent`.                                                                                    |
| `s.tl()` (custom)      | `s.linkTrackVars='eVar1'; s.eVar1='value'; s.tl(this,'o','CTA')` | Define a custom schema containing the relevant properties and track with `trackSelfDescribingEvent`.                                                                                    |
| Props (hit-scoped)     | `s.prop1 = 'value';`                                             | Attach as entities to the specific event.                                                                                                                                               |
| eVars (visit-scoped)   | `s.eVar1 = 'value';`                                             | Attach as entities. For visit-scoped persistence, use [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) or model persistence in the warehouse. |
| Success events         | `s.events = 'event1,purchase';`                                  | Track as specific event types or custom self-describing events. Use entities for associated values.                                                                                     |
| Products string        | `s.products = ';SKU;1;99.99';`                                   | Use built-in ecommerce tracking, or define a `product` entity schema with explicit properties.                                                                                          |
| `s.visitorID`          | `s.visitorID = 'user123';`                                       | Use `setUserId('user123')` to set user ID for all subsequent events.                                                                                                                    |

### Persistence and scope

Adobe Analytics eVars have [configurable persistence](https://experienceleague.adobe.com/en/docs/analytics/components/dimensions/evar#): they can expire after a hit, visit, or specific time period, or persist until a conversion event. This persistence is handled server-side by Adobe.

Snowplow takes an event-centric approach where each event is self-contained. There's no built-in persistence mechanism. Instead, you have two options:

1. **Global context**: attach entities to all events during a session using the tracker's [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) feature
2. **Warehouse modeling**: reconstruct persistence logic in your data models using SQL window functions

For most use cases, warehouse modeling provides more flexibility. You can define custom attribution windows, change persistence logic without re-instrumenting, and apply different persistence rules to historical data.

### Data validation

Adobe Analytics provides Processing Rules and VISTA rules for data transformation, but doesn't validate incoming data against schemas. Invalid or unexpected values are accepted and processed. Classification rules can categorize values post-collection, but there's no prevention of malformed data.

Snowplow requires schematic specification for all data. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out.

Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets. This strict approach ensures high data quality.

You've likely documented your Adobe Analytics implementation in a Solution Design Reference spreadsheet that maps business requirements to props, eVars, and events. Snowplow provides event data management tools for defining and managing tracking plans. Each [tracking plan](/docs/fundamentals/tracking-plans/) contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, use [Snowtype](/docs/event-studio/implement-tracking/) to automatically generate the tracking code snippets.

## Tag management

If you're using Adobe Launch (Adobe Experience Platform Tags) to deploy Adobe Analytics, you'll need to decide how to deploy Snowplow tracking.

There are three main options:

| Approach                                                                                | Description                                            | Best for                                                                                 |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Direct [tracker SDK](/docs/sources/) implementation                                     | Add Snowplow tracker code directly to your application | Teams with engineering resources; cleanest long-term solution                            |
| [Google Tag Manager](/docs/sources/google-tag-manager/)                                 | Use Snowplow's GTM tag templates to deploy tracking    | Teams already using GTM for other tools, or wanting to move away from Adobe Launch       |
| [GTM Server-side](/docs/destinations/forwarding-events/google-tag-manager-server-side/) | Forward events through a server-side container         | Teams already using GTM Server-side, or with strict client-side performance requirements |

Direct SDK implementation is recommended for most migrations. It provides the cleanest separation from your previous Adobe implementation and full access to Snowplow features.

If you choose GTM, you can use Snowplow's tag templates to fire Snowplow events based on data layer pushes, similar to how Adobe Launch rules fire based on data elements.

## Migration phases

We recommend using a parallel-run migration approach. This process can be divided into three phases:

1. Assess and plan
2. Implement and validate
3. Cutover and finalize

### 1. Assess and plan

Your existing Adobe Analytics Solution Design Reference will serve as the foundation for your new Snowplow implementation. Before migration, audit your tracking implementation and Solution Design Reference. Are all the props, eVars, and events still relevant?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Review your existing Adobe Analytics implementation using one or more of these methods:

- Solution Design Reference review: examine your tracking plan spreadsheet for the complete mapping of business requirements to variables
- Code audit: review all `s.t()` and `s.tl()` calls in your codebase, along with variable assignments
- Data Feeds analysis: analyze your Data Feeds exports to identify which props, eVars, and events contain data
- Report suite inspection: use the Adobe Analytics Admin Console to review variable configurations and processing rules

Pay special attention to:

- Variable persistence settings for eVars
- Processing rules that transform incoming data
- VISTA rules for server-side data manipulation
- Classification rules that categorize values
- Calculated metrics that combine multiple variables

You'll need to translate your Adobe Analytics configuration into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which Adobe events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- How should you group props and eVars into logical [entities](/docs/fundamentals/entities/)?
- Which eVar persistence behaviors need to be replicated in warehouse modeling?
- How will you handle the `s.products` string? Snowplow's ecommerce tracking or custom product entities are more flexible alternatives.

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plans into Snowplow is to use the [Snowplow CLI](/docs/api-reference/snowplow-cli/).

> **Info:** The Snowplow CLI includes an [MCP server](/docs/llms-support/cli-mcp-server/) to help you translate your Adobe Analytics event configuration into Snowplow tracking plans.

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical Adobe Analytics Data Feeds in existing tables. Write queries that combine data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical Adobe data into the Snowplow format. This requires a custom engineering project to export Data Feeds, reshape the numbered columns into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset.

### 2. Implement and validate

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Implement and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

Add Snowplow tracking in parallel with your existing Adobe Analytics tracking:

- If you have a web platform, we recommend starting here

  - The [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) is a useful manual testing tool
  - Start with non-critical pages or features

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)

  - Use the Snowplow Inspector to confirm that the tracker is generating the expected events
  - Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate locally
  - Finally, confirm that the tracker can also send events to your warehouse

- Use [Snowtype](/docs/event-studio/implement-tracking/) to generate custom tracking code for your tracking plans

- Test and validate your custom tracking using Micro as before

- Gradually continue this process until you have a complete Adobe Analytics and Snowplow dual tracking implementation

- Gradually roll out tracking to production

> **Info:** [Snowplow Micro](/docs/testing/snowplow-micro/) is a Dockerized local pipeline. It uses the same schema registry as your production pipeline, allowing developers to quickly confirm that the tracked events are valid.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Replicate any eVar persistence logic in your warehouse models
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily page views, or unique visitors

By the end of this phase, you'll have a validated Snowplow implementation that recreates your Adobe Analytics tracking and modeling. The goal is to be confident that the new Snowplow pipeline is capturing all critical business logic correctly.

### 3. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission Adobe Analytics.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the Adobe Analytics tracking from your codebases. Finally, archive your Adobe Analytics configuration and Data Feeds exports.

---

# Google Analytics to Snowplow
> Migrate from Google Analytics to Snowplow, including data layer implementations with Google Tag Manager.
> Source: https://docs.snowplow.io/docs/migration-guides/google-analytics/

This guide is to help technical implementers migrate from Google Analytics to Snowplow.

## Platform differences

There are [significant differences](https://snowplow.io/comparisons/snowplow-vs-google-analytics) between Google Analytics and Snowplow as data platforms. For migration, it's important to understand how Snowplow structures events differently from GA4. This affects both your data layer implementation and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Google Analytics 4                                                                                | Snowplow                                                                                                                                                           |
| ----------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Capturing user behavior | Uses `gtag('event', ...)` with event names, or `dataLayer.push()` with GTM triggers and variables | Direct tracker SDK calls for built-in or custom [event](/docs/fundamentals/events/) types                                                                          |
| Contextual event data   | Included in the event `parameters` object                                                         | Included in the event as reusable [entities](/docs/fundamentals/entities/)                                                                                         |
| Warehouse tables        | BigQuery only; daily partitioned `events_YYYYMMDD` table with nested fields                       | In [most warehouses](/docs/destinations/warehouses-lakes/) one big table with columns for every event or entity; in Redshift, one table per custom event or entity |
| Data validation         | None                                                                                              | All events and entities defined by JSON schemas                                                                                                                    |

### Event structure

The main GA4 tracking method is `gtag('event', ...)` with event names and parameter objects. The event name describes the action taken, and the parameters provide contextual information about that action.

You might have implemented these calls directly in your codebase. More likely, you're using Google Tag Manager (GTM) and tracking behavior via the data layer. When using GTM, your application pushes structured data to the `dataLayer` array. GTM then processes this data using triggers and variables to fire the appropriate `gtag()` calls.

Here's an example GA4 web ecommerce event using the data layer. The application pushes a `purchase` event with an `ecommerce` object containing transaction details:

```javascript
dataLayer.push({

  event: 'purchase',

  ecommerce: {

    transaction_id: '12345',

    value: 25.42,

    currency: 'USD',

    items: [{

      item_id: 'SKU_123',

      item_name: 'Product Name',

      price: 25.42,

      quantity: 1

    }]

  }

});
```

GTM processes this data layer push and fires the corresponding GA4 event. Alternatively, you might have implemented the event directly using `gtag()` calls, like this:

```javascript
gtag('event', 'purchase', {

  transaction_id: '12345',

  value: 25.42,

  currency: 'USD',

  items: [{

    item_id: 'SKU_123',

    item_name: 'Product Name',

    price: 25.42,

    quantity: 1

  }]

});
```

To directly track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event. The tracking code looks similar:

```javascript
snowplow('trackTransaction', {

  transaction_id: 'T_12345',

  revenue: 25.42,

  currency: 'USD',

  products: [{

    id: 'SKU_123',

    name: 'Product Name',

    price: 25.42,

    quantity: 1

  }]

});
```

### Tracking comparison

This table shows how some data layer GA4 tracking maps to Snowplow web tracking:

| GA4 Event     | GA4 Data Layer Example                                                                                                      | Snowplow Implementation                                                                                                                                        |
| ------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `page_view`   | Sent by default by GTM, or `dataLayer.push({event: 'page_view'})`.                                                          | Use `trackPageView`                                                                                                                                            |
| `purchase`    | `dataLayer.push({event: 'purchase', ecommerce: {transaction_id: '123', value: 99.99}})`                                     | Use built-in ecommerce tracking, or define a custom `purchase` schema containing `transaction_id` and `value` properties, and track `trackSelfDescribingEvent` |
| Custom events | Define custom GTM configuration then call e.g. `dataLayer.push({event: 'read_article', article_title: 'Interesting text'})` | Define a custom `read_article` schema containing `article_title` property and track with `trackSelfDescribingEvent`                                            |
| Send user IDs | `dataLayer.push({user_id: 'USER_123'})`, configured as user property in GTM                                                 | Use `setUserId('USER_123')` to set the user ID                                                                                                                 |
| Page metadata | `dataLayer.push({page_type: 'product', category: 'electronics'})`, configured as custom dimensions in GTM                   | Define custom entities for page context and attach to relevant events                                                                                          |

### Warehouse loading and data modeling

The key differences between GA4 and Snowplow events are in the warehouse loading and data modeling.

With GA, the example `purchase` event would be loaded into an `events_YYYYMMDD` table in BigQuery, with nested RECORD fields for parameters and user properties. Each parameter exists as key-value pairs with separate columns for different data types: `string_value`, `int_value`, `float_value`, or `double_value`. Analysts must use complex `UNNEST` operations to extract parameters and join data.

With Snowplow, the example event would be processed as a `transaction` [event](/docs/fundamentals/events/) with one `product` [entity](/docs/fundamentals/entities/). The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser.

In BigQuery, Databricks, or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for the `transaction` event and a column for each entity. In Redshift, each event and entity is loaded into its own table.

You can add the same `product` entity to any other relevant event, such as `add_to_cart` or `view_item`. This simplifies analysis of product lifecycles, from initial product view to purchase.

Snowplow tracker SDKs provide built-in methods for tracking page views, along with many other kinds of events. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) for initial modeling and common analytics use cases.

### Data validation

GA4 is loosely typed and doesn't validate incoming events. You can define expected properties and custom configuration, but events that don't match their specification are still processed. The defined configurations are mainly used for reporting in the GA4 interface.

The lack of validation leads to fragmented and low quality data. For example, the `price` for a purchase event might be accidentally tracked in different places as `25.42`, `"25.42"`, or even `2542`. In the warehouse, this parameter's values will be split over `float_value`, `string_value`, and `int_value` columns, making querying more complex.

Event specification and validation isn't optional for Snowplow. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out.

Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets. This strict approach ensures high data quality.

#### Tracking plans

You've probably defined which GA4 events and parameters to track in a tracking plan spreadsheet or similar document. Snowplow provides event data management tools for defining and managing tracking plans. Each [tracking plan](/docs/fundamentals/tracking-plans/) contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, you may be able to use [Snowtype](/docs/event-studio/implement-tracking/) to automatically generate the tracking code snippets.

## Migration approaches

In general, we recommend a parallel-run migration approach. This involves running both GA4 and Snowplow tracking simultaneously, allowing you to validate your Snowplow data in your warehouse before switching over completely.

If you're using GA4 with direct `gtag()` calls, you can replace these with Snowplow tracker SDK calls in your application code.

However, Snowplow also provides templates and tags for GTM and GTM Server-side. You could choose to migrate your data layer tracking and your modeling implementations in separate stages. This table summarizes the three possible approaches to reach a parallel-run dual tracking implementation:

| Approach                                                                                        | Phase 1                                                              | Phase 2                                                                                         | Phase 3                           | Best for                                                                              |
| ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------- |
| Direct [tracker SDK](/docs/sources/) calls                                                      | Define Snowplow [tracking plans](/docs/fundamentals/tracking-plans/) | Add Snowplow tracking calls; implement Snowplow data models                                     |                                   | Teams with tracking implementation engineer resources; greenfield validation projects |
| [Snowplow GTM templates](/docs/sources/google-tag-manager/)                                     | Define Snowplow [tracking plans](/docs/fundamentals/tracking-plans/) | Configure Snowplow tags to receive data layer events; implement Snowplow data models            | Implement Snowplow tracking calls | Validation of Snowplow data quality for analytics or marketing teams                  |
| [GTM Server-side](/docs/destinations/forwarding-events/google-tag-manager-server-side/) tagging | Define Snowplow [tracking plans](/docs/fundamentals/tracking-plans/) | Use GTM Server-side to forward your existing events to Snowplow; implement Snowplow data models | Implement Snowplow tracking calls | Validation of Snowplow data quality for teams already using GTM Server-side           |

Using Snowplow's GTM templates or tags is a temporary solution to help you migrate.

## Migration phases

We recommend using a parallel-run migration approach. This process can be divided into four phases:

1. Assess and plan
2. Validate data
3. Finish implementation
4. Cutover and finalize

### 1. Assess and plan

Your existing GA4 event configuration will serve as the foundation for your new Snowplow implementation. Before migration, we advise auditing your tracking calls and event configuration. Are all the events and parameters still relevant? Does your configuration match your tracking plan?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Review your existing GA4 implementation using one or more of these methods:

- Tag manager inspection: if using GTM, export your container configuration to identify all GA4 tags, their triggers, and variable mappings
- Programmatic API export: use the [Google Analytics Admin API](https://developers.google.com/analytics/devguides/config/admin/v1) to retrieve the full JSON definition of all custom event configurations
- Code audit: review all `dataLayer.push()` or `gtag('event', ...)` calls in your codebase
- Warehouse inference: analyze your BigQuery `events_YYYYMMDD` tables to identify all event names and parameters in use

For data layer implementations, pay special attention to:

- The structure of your data layer objects and how properties are nested
- Which GTM variables extract and transform data layer properties
- How GTM triggers determine when events are sent
- Any custom JavaScript variables that process data layer data

You'll need to translate your GA4 event configuration into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which GA4 events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- Are there sets of event parameters used in multiple places that could be defined as [entities](/docs/fundamentals/entities/) instead?
- What's the best combination of event properties and entities to capture the same data as GA4?
- If you're using GTM, will you go straight to implementing direct tracker SDK calls, or use Snowplow's GTM templates or GTM Server-side tagging to maintain the data layer pattern initially?

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plans into Snowplow is to use the [Snowplow CLI](/docs/api-reference/snowplow-cli/).

> **Info:** The Snowplow CLI includes an [MCP server](/docs/api-reference/snowplow-cli/#mcp-server) to help you translate your GA4 event configuration into Snowplow tracking plans.

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical GA4 data in BigQuery. Write queries that combine data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical GA4 data into the Snowplow format. This requires a custom engineering project to export GA4 data, reshape it into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset.

### 2. Validate data

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Start sending events to Snowplow and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

> **Info:** Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate your Snowplow events locally, before sending them to your warehouse.
>
> Snowplow Micro is a Dockerized local pipeline. It uses the same schema registry for event validation as your production pipeline.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily event counts, or unique users

Your GA4 implementation will stay live until the final migration phase. By the end of this phase, you'll have a validated Snowplow implementation that captures or recreates your GA4 tracking, and recreates your modeling. The goal is to be confident that the new Snowplow pipeline is capturing critical business logic correctly.

#### Direct tracker SDK calls

For the most straightforward migration, add Snowplow tracking in parallel with your existing GA4 tracking:

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)
- Use the [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) to confirm that the tracker is generating the expected events
- Use [Snowtype](/docs/event-studio/implement-tracking/) to generate custom tracking code for your tracking plans

#### Snowplow GTM templates

If you're using GTM and don't have the resources to implement Snowplow tracking yet, you can use [Snowplow's GTM tag templates](/docs/sources/google-tag-manager/) to generate Snowplow events without changing your application code:

- Keep your existing `dataLayer.push()` calls in your application code
- Install the [Snowplow GTM tag template](/docs/sources/google-tag-manager/) in your GTM web container
- Configure GTM triggers to fire Snowplow tags based on your data layer events
- Use GTM variables to map data layer properties to Snowplow event and entity fields

For example, the existing data layer push stays the same:

```javascript
dataLayer.push({

  event: 'purchase',

  ecommerce: {

    transaction_id: '12345',

    value: 25.42,

    currency: 'USD'

  }

});
```

In GTM, you would configure a Snowplow tag that fires when `event` equals `purchase`, mapping the ecommerce properties to a custom Snowplow self-describing event or using the built-in ecommerce tracking.

#### GTM server-side tagging

If you're already using GTM Server-side to forward events, you could configure Snowplow as a destination using the [Snowplow GTM Server-side tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/):

1. Keep your existing `dataLayer.push()` calls in your application code
2. Configure GTM web container to forward events to GTM server-side
3. Set up Snowplow tracking in the GTM server-side container
4. Use GTM server-side variables and transformations to map data layer data to Snowplow events

### 3. Finish implementation

If you used Snowplow GTM templates or GTM server-side tagging in the previous phase, it's time to implement direct Snowplow tracker SDK calls in your application code.

The benefits of a full Snowplow implementation include:

- Enables a complete migration away from GA4
- Allows you to take advantage of all Snowplow features, including out-of-the-box tracking definitions and event data management tools
- Increased page performance
- Improved debugging and maintenance

### 4. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission GA4.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the GA4 tracking from your codebases. Finally, archive your GA4 configuration and data exports.

---

# Migration guides
> Migrate to Snowplow from other analytics solutions using parallel-run or full re-architecture approaches.
> Source: https://docs.snowplow.io/docs/migration-guides/

This section contains advice for migrating to Snowplow from other solutions. For more advice on tracking plans, check out our [best practices guide](/docs/fundamentals/tracking-design-best-practice/).

When migrating to Snowplow from another analytics solution, there are two possible strategies: parallel-run or full re-architecture.

We recommend a **parallel-run** approach as the lowest-risk strategy. It involves running both systems simultaneously (dual tracking) before switching over to Snowplow entirely. This allows you to test and validate your new Snowplow data in your warehouse, without affecting any existing workflows or production systems.

A **full re-architecture** or "rip-and-replace" approach is faster but riskier, involving a direct switch from your existing system to Snowplow. This is best suited for:

- Major application refactors where the switch can be part of a larger effort
- Teams with high risk tolerance and robust automated testing frameworks
- New projects or applications with minimal legacy systems

A full re-architecture strategy requires thorough testing in a staging environment to prevent data loss.

---

# Migrate from Mixpanel to Snowplow
> Migrate from Mixpanel to Snowplow with guidance on event tracking, user profiles, super properties, and warehouse modeling.
> Source: https://docs.snowplow.io/docs/migration-guides/mixpanel/

This guide is to help technical implementers migrate from Mixpanel to Snowplow.

## Platform differences

There are significant differences between Mixpanel and Snowplow as data platforms. For migration, it's important to be aware of how Snowplow structures events differently from Mixpanel. This affects how you'll implement tracking and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Mixpanel                                                                   | Snowplow                                                                                                                                                            |
| ----------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capturing user behavior | Tracked as named events with properties                                    | Tracked as structured events with built-in types or custom [events](/docs/fundamentals/events/) with validated schemas                                              |
| Contextual event data   | Attached as properties within the event                                    | Attached as separate reusable [entities](/docs/fundamentals/entities/)                                                                                              |
| User and group data     | Tracked separately                                                         | Attached to events as part of the event payload or as reusable entities                                                                                             |
| Warehouse tables        | Events in single or per-event-type tables; user profiles in separate table | In [most warehouses](/docs/destinations/warehouses-lakes/), one big table with columns for every event or entity; in Redshift, one table per custom event or entity |
| Data validation         | Optional Tracking Plans                                                    | All events and entities defined by JSON schemas                                                                                                                     |

### Event structure

Mixpanel tracks user behavior with `track()`. Here's an example custom web ecommerce Mixpanel event. The event name is `Product Purchased`, describing the action taken. The properties object provides contextual information about the specific purchase:

```javascript
mixpanel.track('Product Purchased', {

  product_id: 'SKU_123',

  product_name: 'Widget',

  price: 49.95,

  quantity: 2,

  category: 'Electronics'

});
```

This event loads into Mixpanel's events warehouse table with properties as columns. The table structure depends on your Mixpanel warehouse export configuration.

To track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event. Alternatively, you could define a custom `product_purchased` schema and track it as a [self-describing event](/docs/fundamentals/events/#self-describing-events):

```javascript
snowplow('trackSelfDescribingEvent', {

  event: {

    schema: 'iglu:com.acme/product_purchased/jsonschema/1-0-0',

    data: {

      product_id: 'SKU_123',

      product_name: 'Widget',

      price: 49.95,

      quantity: 2,

      category: 'Electronics'

    }

  }

});
```

The differences are in the warehouse loading and data modeling:

- Snowplow processes this as a self-describing [event](/docs/fundamentals/events/) with validated [schema](/docs/fundamentals/schemas/)

- The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser

- In BigQuery, Databricks, or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for each entity

  - Columns have type `OBJECT` in Snowflake, and `REPEATED RECORD` in BigQuery
  - In Redshift, each event and entity is loaded into its own table

Snowplow tracker SDKs provide built-in methods for tracking page views and screen views, along with many other kinds of events. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling and common analytics use cases.

### Events and users

Mixpanel separates timestamped events from persistent user profiles. Events capture actions with timestamps, while user profiles store persistent attributes like name, email, or subscription tier. Here's the Mixpanel pattern:

```javascript
// Track event

mixpanel.track('Page Viewed');



// Set user profile properties

mixpanel.people.set({

  'name': 'Jane Doe',

  'email': 'jane@example.com',

  'plan': 'Premium'

});
```

In Mixpanel, events and user profiles are typically stored in separate tables. Analysts must join these tables to analyze user behavior with profile attributes.

There's no separate user profile system in Snowplow. You can set a user identifier as part of every event, and attach user attribute entities to events as needed:

```javascript
// Set user ID

snowplow('setUserId', 'user_123');



// Track event with user context entity

snowplow('trackPageView', {

  context: [{

    schema: 'iglu:com.acme/user/jsonschema/1-0-0',

    data: {

      name: 'Jane Doe',

      email: 'jane@example.com',

      plan: 'Premium'

    }

  }]

});
```

This event-centric approach means you can reconstruct user state from the events in the warehouse. This provides a complete audit trail of all attribute changes over time.

### Super properties

In Mixpanel, super properties are event properties that are automatically attached to every event you track. They're used to capture contextual information that applies across all events, such as user plan type, app version, or environment.

```javascript
// Register super properties in Mixpanel

mixpanel.register({

  app_version: '2.1.0',

  environment: 'production',

  user_plan: 'premium'

});



// All subsequent events include these properties

mixpanel.track('Page Viewed');

mixpanel.track('Image Expanded');
```

There's no direct Snowplow equivalent to super properties. Instead, you attach "global context" [entities](/docs/fundamentals/entities/) to every event. This approach provides the same functionality with stronger typing and validation through schemas.

Here's how to implement the same pattern with the [JavaScript tracker](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/):

```javascript
// Define your global context entity

const appContext = {

  schema: 'iglu:com.acme/app_context/jsonschema/1-0-0',

  data: {

    app_version: '2.1.0',

    environment: 'production',

    user_plan: 'premium'

  }

};



// Add as global context - attached to all events

snowplow('addGlobalContexts', [appContext]);



// All subsequent events include this entity

snowplow('trackPageView');

snowplow('trackSelfDescribingEvent', {

  event: {

    schema: 'iglu:com.acme/image_expanded/jsonschema/1-0-0',

    data: { button_id: 'submit' }

  }

});
```

You can also update the global context properties dynamically during runtime. This is useful when values change during the session.

### Tracking comparison

This table shows how example Mixpanel tracking calls could be mapped to Snowplow tracking:

| Mixpanel API Call      | Mixpanel Example                                                         | Snowplow Implementation                                                                                                                                                       |
| ---------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `track()`              | `mixpanel.track('Order Completed', {revenue: 99.99})`                    | Use one of the built-in event types, or define a custom `order_completed` schema containing `revenue` property and track with `trackSelfDescribingEvent`.                     |
| `identify()`           | `mixpanel.identify('user_123')`                                          | Use `setUserId('user_123')` to set user ID for all subsequent events.                                                                                                         |
| `alias()`              | `mixpanel.alias('user_123', 'anon_456')`                                 | No direct equivalent. Track identity changes as custom events with `user_alias_created` schema if needed for analysis. Use `setUserId` to update the current user identifier. |
| `register()`           | `mixpanel.register({plan: 'premium'})`                                   | Attach as entities to all events.                                                                                                                                             |
| `.people.set()`        | `mixpanel.people.set({email: 'user@example.com'})`                       | Attach custom user entities to events.                                                                                                                                        |
| `.people.increment()`  | `mixpanel.people.increment('page_views')`                                | No direct equivalent. Track events and calculate aggregates in warehouse.                                                                                                     |
| `.track_with_groups()` | `mixpanel.track_with_groups('Button Clicked', {groupId: 'company-123'})` | Attach custom group entities to events containing group identifiers and relevant group properties.                                                                            |
| `reset()`              | `mixpanel.reset()`                                                       | Clear user ID with `setUserId(null)` or equivalent, depending on tracker. The tracker will generate a new anonymous identifier.                                               |

### Data validation

You've presumably defined Tracking Plans for your Mixpanel implementation. The suggested Mixpanel approach is to create and maintain an external spreadsheet of events and properties to track. Within Mixpanel, the Lexicon feature allows you to manage definitions, and hide or drop events. There's no built-in validation of tracked events.

Snowplow requires schematic specification for all data. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out.

To help maintain high data quality, Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets.

Snowplow includes in-product management and tools for [tracking plans](/docs/fundamentals/tracking-plans/). Each tracking plan contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, use [Snowtype](/docs/event-studio/implement-tracking/) to automatically generate the tracking code snippets.

> **Info:** The Snowplow CLI includes an [MCP server](/docs/llms-support/cli-mcp-server/) to help you translate your Mixpanel Tracking Plans into Snowplow tracking plans.

## Migration phases

Use a parallel-run migration approach. This process can be divided into three phases:

1. Assess and plan
2. Implement and validate
3. Cutover and finalize

### 1. Assess and plan

Use your existing Mixpanel Tracking Plans as the foundation for your new Snowplow implementation. Before migration, audit your tracking calls and Tracking Plans. Are all the events and properties still relevant?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Export your Tracking Plans using one of these methods:

- Programmatic API export (recommended): use the [Mixpanel Public API](https://developer.mixpanel.com/reference/overview) to retrieve the full definition of each Tracking Plan
- Manual CSV download: download a CSV file from the Mixpanel UI for a less detailed but quick inventory of your events and properties
- Warehouse inference: if you no longer have access to your Mixpanel account, you can infer the events and properties from your warehouse data

You'll need to translate your Mixpanel Tracking Plans into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which Tracking Plan events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- How should you migrate Mixpanel super properties? These are typically best represented as global context [entities](/docs/fundamentals/entities/) that are attached to all events.
- Which user profile properties should be captured as user entities? You might not need to attach user entities to every event, only those where the user context is relevant for analysis.
- Which events use group analytics? You'll need to define group entities for these.
- Are there sets of event properties used in multiple places that could be defined as reusable entities instead?

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plan tracking plans into Snowplow is to use the [Snowplow CLI](/docs/api-reference/snowplow-cli/).

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical Mixpanel data in existing tables. Write queries that combine data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical Mixpanel data into the Snowplow format. This requires a custom engineering project to export Mixpanel data, reshape it into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset. Note that this is more complex for Mixpanel than other platforms due to the separation of events and user profiles.

### 2. Implement and validate

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Implement and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

Add Snowplow tracking in parallel with your existing Mixpanel tracking:

- If you have a web platform, start here

  - The [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) is a useful manual testing tool
  - Start with non-critical pages or features

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)

  - Use the Snowplow Inspector to confirm that the tracker is generating the expected events
  - Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate locally
  - Finally, confirm that the tracker can also send events to your warehouse

- Use [Snowtype](/docs/event-studio/implement-tracking/) to generate custom tracking code for your tracking plans

- Test and validate your custom tracking using Micro as before

- Gradually continue this process until you have a complete Mixpanel and Snowplow dual tracking implementation

- Gradually roll out tracking to production

> **Info:** [Snowplow Micro](/docs/testing/snowplow-micro/) is a Dockerized local pipeline. It uses the same schema registry as your production pipeline, allowing developers to quickly confirm that the tracked events are valid.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily event counts, or unique users

By the end of this phase, you'll have a validated Snowplow implementation that recreates your Mixpanel tracking and modeling. The goal is to be confident that the new Snowplow pipeline is capturing all critical business logic correctly.

### 3. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission Mixpanel.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the Mixpanel tracking from your codebases. Finally, close your Mixpanel account.

---

# Segment to Snowplow
> A guide for technical implementers migrating from Segment to Snowplow, covering event structure differences, tracking plan translation, and a phased migration approach.
> Source: https://docs.snowplow.io/docs/migration-guides/segment/

This guide is to help technical implementers migrate from Segment to Snowplow.

## Platform differences

There are a [number of differences](https://snowplow.io/comparisons/snowplow-vs-segment) between Segment and Snowplow as a data platform. For migration, it's important to be aware of how Snowplow structures events differently from Segment. This affects how you'll implement tracking and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Segment                                   | Snowplow                                                                                                                                                            |
| ----------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capturing user behavior | Uses `track` events                       | Choose from the built-in event types, or use `trackSelfDescribingEvent` to track a custom [event](/docs/fundamentals/events/)                                       |
| Contextual event data   | Included in the event `properties` object | Included in the event as reusable [entities](/docs/fundamentals/entities/)                                                                                          |
| User identity           | Tracked separately as an `identify` event | Included in the event as a reusable entity                                                                                                                          |
| Warehouse tables        | One table per custom event type           | In [most warehouses](/docs/destinations/warehouses-lakes/), one big table with columns for every event or entity; in Redshift, one table per custom event or entity |
| Data validation         | Optional Tracking Plan                    | All events and entities defined by JSON schemas                                                                                                                     |

### Event structure

Segment's core method for tracking user behavior is `track`. Here's an example custom web ecommerce Segment event. The event name is `Transaction Completed`, describing the action taken. The properties object provides contextual information about the specific transaction, including an array of product objects:

```javascript
analytics.track('Transaction Completed', {

  transaction_id: 'T_12345',

  revenue: 99.99,

  currency: 'USD',

  products: [{

    id: 'ABC123',

    name: 'Widget',

    price: 99.99,

    quantity: 1

  }]

})
```

This event would be loaded into a `Transaction Completed` warehouse table. Analysts must `UNION` tables together to reconstruct user journeys.

To track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event. The tracking code looks very similar:

```javascript
snowplow('trackTransaction', {

  transaction_id: 'T_12345',

  revenue: 99.99,

  currency: 'USD',

  products: [{

    id: 'ABC123',

    name: 'Widget',

    price: 99.99,

    quantity: 1

  }]

})
```

The differences are in the warehouse loading and data modeling:

- Snowplow processes this as a `transaction` [event](/docs/fundamentals/events/) with one `product` [entity](/docs/fundamentals/entities/)

- The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser

- In BigQuery or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for the `transaction` event and a column for each entity

  - Columns have type `OBJECT` in Snowflake, and `REPEATED RECORD` in BigQuery
  - In Redshift, each event and entity is loaded into its own table

You can add the same `product` entity to any other relevant event, such as `add_to_cart` or `view_product`. This simplifies analysis of product lifecycles, from initial product view to purchase.

Snowplow tracker SDKs provide built-in methods for tracking page views and screen views, along with many other kinds of events. For example, button clicks, form submissions, page pings (activity), or media interactions. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling and common analytics use cases.

### Tracking comparison

This table shows how example Segment tracking calls could be mapped to Snowplow tracking:

| Segment API Call | Segment Example                                                 | Snowplow Implementation                                                                                                                                                      |
| ---------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `track()`        | `track('Order Completed', {revenue: 99.99, currency: 'USD'})`   | Use one of the built-in event types, or define a custom `order_completed` schema containing `revenue` and `currency` properties, and track `trackSelfDescribingEvent`.       |
| `page()`         | `page('Pricing')`                                               | Use `trackPageView`. The tracker SDK will automatically capture details such as `title` and `url`.                                                                           |
| `screen()`       | `screen('Home Screen')`                                         | Use `trackScreenView`.                                                                                                                                                       |
| `identify()`     | `identify('user123', {plan: 'pro', created_at: '2024-01-15'})`  | Call `setUserId('user123')` to track the ID in all events. Attach a custom `user` entity with schema containing `plan` and `created_at` properties.                          |
| `group()`        | `group('company-123', {name: 'Acme Corp', plan: 'Enterprise'})` | No direct equivalent. Attach a custom `group` entity to your events, or track group membership changes as custom events with `group_joined` or `group_updated` schemas.      |
| `alias()`        | `alias('new-user-id', 'anonymous-id')`                          | No direct equivalent. Track identity changes as custom events with `user_alias_created` schema. Use `setUserId` to update the current user identifier for subsequent events. |

### Data validation

You've presumably defined Tracking Plans for your Segment implementation. The suggested Segment approach for creating a Tracking Plan is to first track some events, then use those events as the basis for a template. You choose how to handle future validation failures: filter out events that don't match their specification, allow with warnings, or pass after autocorrecting minor issues. Using a Tracking Plan is optional.

Schematic event specification isn't optional for Snowplow. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out. There's no pass-with-warning or autocorrection feature.

Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets. This strict approach ensures high data quality.

Snowplow also provides event data management tools for defining and managing tracking plans. Each Snowplow [tracking plan](/docs/fundamentals/tracking-plans/) contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, use [Snowtype](/docs/event-studio/implement-tracking/) to automatically generate the tracking code snippets.

## Migration phases

We recommend using a parallel-run migration approach. This process can be divided into three phases:

1. Assess and plan
2. Implement and validate
3. Cutover and finalize

### 1. Assess and plan

Your existing Segment Tracking Plans will serve as the foundations for your new Snowplow implementation. Before migration, we advise auditing your tracking calls and Tracking Plans. Are all the events and properties still relevant?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Export your Tracking Plans using one of these methods:

- Programmatic API export (recommended): use the [Segment Public API](https://segment.com/docs/api/public-api/) to retrieve the full JSON definition of each Tracking Plan
- Manual CSV download: download a CSV file from the Segment UI for a less nested but potentially incomplete inventory of your events and properties
- Warehouse inference: if you no longer have access to your Segment account, you can infer the events and properties from your warehouse data

You'll need to translate your Segment Tracking Plans into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which Tracking Plan events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- Are there sets of event properties used in multiple places that could be defined as [entities](/docs/fundamentals/entities/) instead?
- What's the best combination of event properties and entities to capture the same data as the non-`track` Segment events?

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plans into Snowplow is to use the [Snowplow CLI](/docs/api-reference/snowplow-cli/).

> **Info:** The Snowplow CLI includes an [MCP server](/docs/api-reference/snowplow-cli/#mcp-server) to help you translate your Segment Tracking Plans into Snowplow tracking plans.

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical Segment data in existing tables. Write queries that `UNION` data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical Segment data into the Snowplow format. This requires a custom engineering project to export Segment data, reshape it into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset.

### 2. Implement and validate

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Implement and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/api-reference/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

Add Snowplow tracking in parallel with your existing Segment tracking:

- If you have a web platform, we recommend starting here

  - The [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) is a useful manual testing tool
  - Start with non-critical pages or features

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)

  - Use the Snowplow Inspector to confirm that the tracker is generating the expected events
  - Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate locally
  - Finally, confirm that the tracker can also send events to your warehouse

- Use [Snowtype](/docs/event-studio/implement-tracking/) to generate custom tracking code for your tracking plans

- Test and validate your custom tracking using Micro as before

- Gradually continue this process until you have a complete Segment and Snowplow dual tracking implementation

- Gradually roll out tracking to production

> **Info:** [Snowplow Micro](/docs/testing/snowplow-micro/) is a Dockerized local pipeline. It uses the same schema registry as your production pipeline, allowing developers to quickly confirm that the tracked events are valid.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily event counts, or unique users

By the end of this phase, you'll have a validated Snowplow implementation that recreates your Segment tracking and modeling. The goal is to be confident that the new Snowplow pipeline is capturing all critical business logic correctly.

### 3. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission Segment.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the Segment tracking from your codebases. Keep the Segment sources active for a short time as a final fallback before fully decommissioning. Finally, close your Segment account.

---

# Automatically generated data models
> Generate optimized, analysis-ready data models directly from your tracking plans for views or incremental dbt models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/automatically-generated-data-models/

Automatic data model generation allows you to generate optimized models directly from your [tracking plans](/docs/event-studio/tracking-plans/) in [Snowplow Console](https://console.snowplowanalytics.com). You can use the models directly in your warehouse, or integrate them into existing dbt projects. The models are ready for analysis, BI tool, or reverse ETL use cases.

The autogenerated data models filter for only the events of interest, while flattening your event and entity data structures into a column for each property.

> **Note:** Automatically generated data models is not supported for Redshift.

## Key features

Choose exactly how you want the model to look, based on your use case.

### Configurable events and entities

You can specify which parts of the tracking plan to include:

- Event specifications
- Entities
- [Atomic Snowplow event properties](/docs/fundamentals/canonical-event/#common-fields)

### Automatic data flattening

To create a single wide table, the tool automatically flattens data structures into individual columns.

For event data structures, all properties are flattened into their own columns.

For entities, flattening depends on cardinality:

- Single entities, such as `user`, are flattened into separate columns, e.g., `user_id`, `user_email`
- Multiple entities, such as a `products` array, are stored as a single array column that you can unnest later

### Event specification inference

If you're using [Snowtype](/docs/event-studio/implement-tracking/) to generate your tracking code, the model generator can filter for event specification IDs, using the attached event specification entity. This guarantees that all events were sent with the intent to implement this tracking plan, and ensures high data quality.

If you're not using Snowtype, the generated models perform more complex event filtering:

- Models include all rows matching the tracking plan definition
- Use "best effort" matching based on events, entities, cardinalities, and rules

These no-Snowtype models may include some unintended events that aren't part of the tracking plan. However, this approach allows access to historical data, and requires no adjustments to existing tracking implementations.

### Deployment options

You can choose to create a view or an incremental dbt model.

**Views** are best for lightweight use cases when you need immediate access to data, with minimal setup.

View statements can be run directly in your data warehouse without the need for additional tools. We recommend using views when your atomic events table is small, or when you won't be regularly querying the data. When using a view, ensure you reference the partition key of the atomic table to ensure your queries are performant and cost efficient.

**Incremental dbt models** are ideal for production pipelines when you want to materialize data efficiently, and integrate it into existing dbt projects. We recommend using incremental models when your atomic events table is large, or when you'll be regularly querying the data.

Choose to create a simple model without dependencies, or one based on the Snowplow [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) or [Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) packages:

- **Simple** models are compatible with any dbt project with no package dependencies. These models will materialize all data for the model on the first run, while processing only new events for every subsequent run.
- **Models based on [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) or [Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/)** are designed to work with our existing dbt packages. They build on `events_this_run` tables for efficient incremental processing.

This table compares the different options:

| Consideration             | View                                        | Simple incremental                           | Unified Digital or Normalize incremental   |
| ------------------------- | ------------------------------------------- | -------------------------------------------- | ------------------------------------------ |
| **Best for**              | Real-time data access, exploratory analysis | Standalone projects, new dbt implementations | Existing Snowplow dbt package users        |
| **Dependencies**          | None                                        | None                                         | Requires Snowplow dbt packages             |
| **Setup complexity**      | Minimal                                     | Minimal                                      | Moderate (variable configuration required) |
| **Processing efficiency** | Query-time processing                       | Standard incremental                         | Optimized with `events_this_run` tables    |
| **Integration**           | Independent operation                       | Independent operation                        | Coordinated with existing Snowplow models  |
| **Storage requirements**  | No additional storage                       | Materializes data                            | Materializes data                          |
| **Query performance**     | Depends on underlying data volume           | Fast (pre-computed)                          | Fast (pre-computed)                        |
| **Data freshness**        | Real-time                                   | Batch incremental updates                    | Batch incremental updates                  |

## Generating models

To generate a data model, find the **Data models** tab in [Console](https://console.snowplowanalytics.com/) > **Data collection** > **Tracking plans** > tracking plan details page.

![Data models tab on the example\_tracking\_plan page showing a three-step flow (Your tracking plan data → Generate your data model → Simplified modeled data) and a \&quot;Generate data model\&quot; button](/assets/images/landing-page-0a19d14dfa36fd6c3a7fbeaf3438443b.png)

Follow the instructions to create your model:

- Provide basic information such as name and warehouse destination
- Choose which event specifications, entities, and columns to include
- Confirm if you're using Snowtype
- Choose which kind of model to create

Once you've completed these steps, click **Save to tracking plan** to save the model. You can also download the model now.

![Finalize step of the Generate data model wizard showing two deployment options — \&quot;Create a view\&quot; (selected) and \&quot;Create an incremental dbt model\&quot; — with a SQL preview and a Download button](/assets/images/finalize-9d2d8bdb83c70a9c52fc3dfd1ef2f197.png)

You can view the SQL at any time by choosing **View data model code** from the `⋮` menu.

![Generated data models list showing example\_model as a View on Snowflake prod1, generated on 29 September 2025, with a menu open showing \&quot;View data model code\&quot; and \&quot;Delete data model\&quot; options](/assets/images/created-823a530fa28566660f873afb4570ca69.png)

## Deploying models

How to deploy the model depends on the kind of model you've generated. The first step is the same for all: download the model from Console.

### Views

Deployment steps:

1. Configure the model schema to specify where to create the view
2. Run the model to create the view in your warehouse

### Simple incremental

Deployment steps:

1. Add the SQL file to your dbt project's models directory
2. Run the model

The initial run will materialize all historical data from your `atomic` events table. Subsequent runs will process only new events since the last run timestamp.

### Unified Digital or Normalize incremental

Deployment steps:

1. Add the SQL file to your dbt project's models directory

2. Configure dbt project variables:

   - Set `snowplow__start_date` in your dbt project variables
   - Configure `snowplow__backfill_limit_days` to control the volume of data processed per run

3. Run the model iteratively until the new model catches up to your existing models

4. Monitor progress through dbt logs and manifest updates

New models begin processing from `snowplow__start_date` since they're not yet tracked in the Snowplow Manifest table.

The model won't run if the required dependency isn't already installed in your dbt project.

---

# Modeling data
> An overview of the purpose of data modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/

At Snowplow, we define data modeling as the process of using business logic to, transform, aggregate, or otherwise process event level data. It occurs in the data warehouse, and allows you to extract value from your events data.

The Snowplow atomic data acts as an immutable log of all the actions that occurred across your digital products. The data model takes that data and transforms it into a set of derived tables optimized for analysis.

It is at the data modeling stage that you add your business logic to your Snowplow data; for example, how you define your marketing channels, what counts as a conversion, or how you segment your customers. As this logic is separate from the data you collect, you are able to update the logic and rerun it over your entire set of raw, immutable data to produce new derived data if you change your mind at a later stage. Not only can you add business logic to the data during modeling, you can also aggregate the same data into different tables based on the use cases the different teams in your organization have. Therefore, data modeling is a crucial step towards deriving insights from your data.

Snowplow provides [multiple dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/) to get you started modeling your data.

---

# Configure the Attribution data model
> Configure the Attribution dbt package for marketing attribution analysis with customizable conversion paths and channel groupings.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/attribution/

This page helps you configure the Snowplow Attribution dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `conversion_path_source` | The source (schema and table) of the  paths (touchpoints). By default it is the derived `snowplow_unified_views` table. Optionally it may be hardcoded with a string reference instead of a source with a schema.table or database.schema.table format. | `{{ source('derived', 'snowplow_unified_views') }}` |
| `conversions_source` | The source (schema and table) of your conversion events,  likely the conversions table generated by the Unified package. Optionally it may be hardcoded with a string reference instead of a source with a schema.table or database.schema.table format. | `{{ source('derived', 'snowplow_unified_conversions') }}` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `spend_source` | The source (schema and table) of your marketing spend source. Optional, needed for the ROAS calculation of the snowplow_attribution_overview. Should be changed to a table reference with `spend` by `channel` and/or `campaign` by `spend_tstamp` (which denotes a timestamp field) information. | `not defined` |
| `user_mapping_source` | The source (schema and table) of the user mapping table produced by the unified package. In case the user mapping table is not available, the paths_to_conversion macro needs to be overwritten in the dbt project where the package is referenced. | `{{ source('derived', 'snowplow_unified_user_mapping') }}` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `attribution_start_date` | The date to start processing events from in the package on first run or a full refresh, based on the cv_tstamp (conversion timestamp). | `2023-01-01` |
| `path_lookback_days` | Restricts the model to marketing channels within this many days of the conversion (values of 30, 14 or 7 are recommended). | `30` |
| `path_lookback_steps` | The limit for the number of marketing channels to look at before the conversion. | `0 (unlimited)` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |
| `conversion_clause` | A string of sql to filter on certain conversion events. | `cv_value > 0 and ev.user_identifier is not null` |
| `conversion_window_days` | The last complete nth number of days (calculated from the last processed pageview within page_views_source) to dynamically update the conversion_window_start_date and end_date with. Will only apply if both variables are left as an empty string. | `30` |
| `conversion_window_end_date` | The end date in UTC for the window of conversions to include. It is only used for the drop and recompute report tables/views. | `""` |
| `conversion_window_start_date` | The start date in UTC for the window of conversions to include. It is only used for the drop and recompute report tables/views. | `current_date()-31` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `attribution_list` | List of attribution types to use for reporting. Can be at least one of: first_touch, last_touch, linear, position_based). | `['first_touch', 'last_touch', 'linear', 'position_based']` |
| `campaigns_to_exclude` | List of campaigns to exclude from analysis (empty to keep all campaigns). | `[ ] (no filter applied)` |
| `campaigns_to_include` | List of campaigns to include in the analysis (empty to keep all campaigns). | `[ ] (no filter applied)` |
| `channels_to_exclude` | List of channels to exclude from analysis (empty to keep all channels). For example, users may want to exclude the `Direct` channel from the analysis. | `[ ] (no filter applied)` |
| `channels_to_include` | List of channels to include in the analysis (empty to keep all channels). For example, users may want to include the `Direct` channel only in the analysis. | `[ ] (no filter applied)` |
| `consider_intrasession_channels` | If `false`, only considers the channel at the start of the session (i.e. first page view). If `true`, considers multiple channels in the conversion session as well as historically. | `true` |
| `conversion_hosts` | `url_hosts` to filter to in the data processing | `[] (no filter applied)` |
| `conversion_stitching` | This should be set to true if both the snowplow__conversion_stitching and snowplow__view_stitching variables are also enabled in the Unified package. If allowed it will consider the stitched_user_id field, not the user_identifier in the source data for more accurate results. | `false` |
| `enable_paths_to_non_conversion` | If `true`, enable the paths_to_non_conversion model, which is a drop and recompute table that may be needed for more in-depth attribution analysis (used in the `path_summary` table as well) | `false` |
| `enable_attribution_overview` | If `true`, enable the attribution_overview model, which is a view that creates a great source for BI tools to use for reporting | `true` |
| `path_transforms` | Dictionary of path transforms (and their argument) to perform on the conversion path (see the transform path options in our [model docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)). For `remove_if_not_all` and `remove_if_last_and_not_all` transformations, the argument has to be a non-empty array, for the others it should be null. | `{'exposure_path': null}` |


---

# Configure the Ecommerce data model
> Configure the Ecommerce dbt package for modeling transactions, carts, checkouts, and product performance.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/

This page helps you configure the Snowplow Ecommerce dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.1)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `number_checkout_steps` | The index of the checkout step which represents a completed transaction. This is required to enable working checkout funnel analysis, and has a default value of 4. | `4` |
| `categories_separator` | The separator used to split out your subcategories from your main subcategory. If for example your category field is filled as follows: `books/fiction/magical-fiction` then you should specify `"/"` as the separator in order for the subcolumns to be properly parsed. | `/` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |
| `number_category_levels` | The **maximum** number of levels (depth) of subcategories that exist on your website for products. These subcategories are recorded in the category field of the product context, and should be separated using the separator which is defined below. For example, `books/fiction/magical-fiction` has a level of 3. The value is the number of columns that will be generated in the product tables created by this Snowplow dbt package. Please note that some products can have less than the maximum number of categories specified. | `4` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `enable_mobile_events` | A boolean whether to use the mobile contexts for mobile e-commerce events in the processing (based on the client session and screen view context). | `false` |
| `use_product_quantity` | Flag to use the product quantity value for the number of products in a transaction, instead of the count of events. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `ecommerce_event_names` | The list of event names that the Snowplow e-commerce package will filter on when extracting events from your atomic events table. If you have included any custom e-commerce events, feel free to add their event name in this list to include them in your data models. | `['snowplow_ecommerce_action']` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |
| `session_timestamp` | Determines which timestamp to use for optimizing the queries in our sessionization logic. This should be set to the same timestamp as the field you partition your events table on. This does not affect how sessions are processed and all events will be processed as described in our [sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing) logic. | `collector_tstamp` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_ecommerce_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `carts_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `add_to_cart`, `remove_from_cart`, or `transaction` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `checkout_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `transaction` or `checkout_step` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `disable_ecommerce_carts` | Flag to exclude the `Snowplow E-commerce` [cart entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#cart-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_checkouts` | Flag to exclude the `Snowplow E-commerce` [checkout entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#checkout-step-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_page_context` | Flag to exclude the `Snowplow E-commerce` [page entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#ecommerce-page-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_products` | Flag to exclude the `Snowplow E-commerce` [product entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#product-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_transactions` | Flag to exclude the `Snowplow E-commerce` [transaction entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#transaction-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_user_context` | Flag to exclude the `Snowplow E-commerce` [user entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#ecommerce-user-entity) context in case this is disabled or not used in your tracking. | `false` |
| `products_passthroughs` | Field(s) to carry through from the events table to the derived table. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the first event record in the session based on `derived_tstamp` then `dvce_created_tstamp`. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `transaction_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `transaction` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Identities data model
> Configure the Snowplow Identities dbt package for identity resolution and identifier mapping.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/identities/

This page helps you configure the Snowplow Identities dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.1.0)

This package sets variables to recommended defaults for optimal model performance. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `events` | This is used internally by the package to reference your events table based on other variable values and should not be changed. | `{{ source('atomic', 'events') }}` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from on first run or a full refresh, based on `derived_tstamp`. | `2020-01-01` |
| `identifiers` | The list of identifiers to extract from events and link to each `snowplow_id`. Each entry has a `reference` (the column name in your events table) and an `alias` (the name used in the output tables). Add any event-level fields you want to track as identifiers, such as `email` or `phone`. | `[{'reference': 'domain_userid', 'alias': 'domain_userid'}...` |
| `hash_identifiers` | When set to `true`, all identifier values stored in `snowplow_identities_identifier_mapping` are hashed using SHA-256 before being written to the table. Recommended when identifiers contain PII. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum number of days of new data to be processed since the latest event processed. | `30` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |

### Contexts, filters, and logs




---

# Configure Snowplow dbt packages
> Configure Snowplow dbt packages, including warehouse-specific settings for Postgres, Spark, Databricks, and BigQuery.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/

This page details general configurations that can apply across many of our packages. Each package has specific configuration variables that define how the models run, please see each child page for the specifics of each package.

## Variables

> **Tip:** Do not copy the project yaml from the package into your main project, this can lead to issues. Only add variables and model configurations that you wish to change from the default behavior. Each configuration page contains a UI to help to build your variables.

> **Tip:** When using multiple dbt packages you must be careful to specify which scope a variable or configuration is defined within. In general, always specify each value in your `dbt_project.yml` nested under the specific package e.g.
>
> ```yml
> vars:
>
>   snowplow_web:
>
>     snowplow__atomic_schema: schema_with_snowplow_web_events
>
>   snowplow_mobile:
>
>     snowplow__atomic_schema: schema_with_snowplow_mobile_events
> ```
>
> You can read more about variable scoping in dbt's docs around [variable precedence](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#variable-precedence).

## Warehouse specific configurations

### Postgres

To optimism performance of large Postgres datasets you can create [indexes](https://docs.getdbt.com/reference/resource-configs/postgres-configs#indexes) in your dbt model config for columns that are commonly used in joins or where clauses. For example:

```yaml
# snowplow_web_sessions_custom.sql

{{

  config(

    ...

    indexes=[\{'columns': ['domain_sessionid'], 'unique': True}]

  )

}}
```

### Spark

For Spark environments, Iceberg is currently the supported file format for external tables. We have successfully tested this setup using both Glue and Thrift as connection methods. To use these models, create an external table from the Iceberg lake format in Spark and point your dbt model to this table.

Here's an example profiles.yml configuration for Spark using Thrift:

```yaml
spark:

  type: spark

  host: localhost

  method: thrift

  port: 10000

  schema: default
```

In your dbt\_project.yml, the file\_format is set to `iceberg` by default for Spark. While you can override this in your project's dbt YAML file to use a different file format, please note that Iceberg is currently the only officially supported format.

### Databricks

You can connect to Databricks using either the `dbt-spark` or the `dbt-databricks` connectors. The `dbt-spark` adapter does not allow dbt to take advantage of certain features that are unique to Databricks, which you can take advantage of when using the `dbt-databricks` adapter. Where possible, we would recommend using the `dbt-databricks` adapter.

#### Unity Catalog support

With the rollout of Unity Catalog (UC), the `dbt-databricks` adapter has added support in dbt for the three-level-namespace as of `dbt-databricks>=1.1.1`. As a result of this, we have introduced the `snowplow__databricks_catalog` variable which should be used **if** your Databricks environment has UC enabled, and you are using a version of the `dbt-databricks` adapter that supports UC. The default value for this variable is `hive_metastore` which is also the default name of your UC, but this can be changed with the `snowplow__databricks_catalog` variable.

Since there are many different situations, we've created the following table to help guide your setup process (this should help resolve the `Cannot set database in Databricks!` error):

|                                             | Adapter supports UC and UC Enabled                                                                    | Adapter supports UC and UC not enabled          | Adapter does not support UC                                                                          |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Events land in default `atomic` schema      | `snowplow__databricks_catalog = '\{name_of_catalog}'`                                                 | Nothing needed                                  | `snowplow__databricks_catalog = 'atomic'`                                                            |
| Events land in custom schema (not `atomic`) | `snowplow__atomic_schema = '\{name_of_schema}'` `snowplow__databricks_catalog = '\{name_of_catalog}'` | `snowplow__atomic_schema = '\{name_of_schema}'` | `snowplow__atomic_schema = '\{name_of_schema}'` `snowplow__databricks_catalog = '\{name_of_schema}'` |

#### Optimization of models

The `dbt-databricks` adapter allows our data models to take advantage of the auto-optimization features in Databricks. If you are using the `dbt-spark` adapter, you will need to manually alter the table properties of your derived and manifest tables using the following command after running the data model at least once. You will need to run the command in your Databricks environment once for each table, and we would recommend applying this to the tables in the `_derived` and `_snowplow_manifest` schemas:

```SQL
ALTER TABLE \{TABLE_NAME} SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true, delta.autoOptimize.autoCompact = true);
```

### BigQuery

As mentioned in the [Quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/), many of our packages allow you to specify which column your events table is partitioned on using the `snowplow__session_timestamp` variable. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:

```yml
vars:

  snowplow_mobile:

    snowplow__derived_tstamp_partitioned: false
```

---

# Configure the Fractribution data model
> Configure the legacy Fractribution dbt package for marketing attribution analysis. Superseded by the Attribution package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/fractribution/

> **Note:** The Fractribution Package is no longer maintained, please refer to the [Attribution package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) for marketing attribution analysis with Snowplow

This page helps you configure the Snowplow Fractribution dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.3.6)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `web_user_mapping_table` | The schema and table name of the snowplow web user mapping table, if different to default. | `derived.snowplow_web_user_mapping` |
| `conversions_source` | The source (schema and table) of your conversion events,  likely the conversions table generated by the Unified package. | `{{ source('atomic', 'events') }}` |
| `page_views_source` | The source (schema and table) of the derived `snowplow_web_page_views` table. | `{{ source('derived', 'snowplow_web_page_views') }}` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `path_lookback_days` | Restricts the model to marketing channels within this many days of the conversion (values of 30, 14 or 7 are recommended). | `30` |
| `path_lookback_steps` | The limit for the number of marketing channels to look at before the conversion. | `0 (unlimited)` |
| `conversion_window_days` | The last complete nth number of days (calculated from the last processed pageview within page_views_source) to dynamically update the conversion_window_start_date and end_date with. Will only apply if both variables are left as an empty string. | `30` |
| `conversion_window_end_date` | The end date in UTC for the window of conversions to include. | `""` |
| `conversion_window_start_date` | The start date in UTC for the window of conversions to include. | `current_date()-31` |
| `conversions_source_filter` | A timestamp field the conversion source field is partitioned on (ideally) for optimized filtering, when left blank `derived_tstamp` is used. | `blank` |
| `conversions_source_filter_buffer_days` | The number of days to extend the filter. | `1` |
| `use_snowplow_web_user_mapping_table` | `true` if you are using the Snowplow web model for web user mappings (`domain_userid` => `user_id`) | `false` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `channels_to_exclude` | List of channels to exclude from analysis (empty to keep all channels). For example, users may want to exclude the `Direct` channel from the analysis. | `[ ] (no filter applied)` |
| `channels_to_include` | List of channels to include in the analysis (empty to keep all channels). For example, users may want to include the `Direct` channel only in the analysis. | `[ ] (no filter applied)` |
| `consider_intrasession_channels` | If `false`, only considers the channel at the start of the session (i.e. first page view). If `true`, considers multiple channels in the conversion session as well as historically. | `false` |
| `conversion_hosts` | `url_hosts` to filter to in the data processing | `[] (no filter applied)` |
| `path_transforms` | Dictionary of path transforms (and their argument, `null` if none) to perform on the full conversion path (see [udfs.sql file](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/path_transformations/create_udfs.sql)). | `{'exposure_path': null}` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `run_python_script_in_snowpark` | A flag for if you wish to run the python scripts using Snowpark. | `false` | Snowflake |
| `attribution_model_for_snowpark` | The attribution model to use when running in Snowpark; one of `shapley`, `first_touch`, `last_touch`, `position_based`, `linear`. See the [package docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-fractribution-data-model/#attribution-models) for more information. | `shapley` | Snowflake |


---

# Configure the legacy dbt packages
> Configuration for legacy Snowplow dbt packages including Web, Mobile, and Fractribution that have been superseded by newer packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/

> **Note:** Legacy packages are no longer under active development and are in some cases no longer supported. Please look for a suitable replacement for these packages where possible.

| Package name  | Purpose                                             | Superseded by                                                                                              |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Web           | Transform raw web event data into derived tables    | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Mobile        | Transform raw mobile event data into derived tables | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Fractribution | Marketing attribution analysis                      | [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) |

---

# Configure the Mobile data model
> Configure the legacy Mobile dbt package for mobile event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/mobile/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

This page helps you configure the Snowplow Mobile dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `sessions_table` | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. `{{ ref(\'snowplow_mobile_sessions_custom\') }}`. Please see the [README](https://github.com/snowplow/dbt-snowplow-mobile/tree/main/custom_example) in the `custom_example` directory for more information on this sort of implementation. | `"{{ ref( \'snowplow_mobile_sessions\' ) }}"` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "contexts_com_snowplowanalytics_snowplow_cli...` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "contexts_com_snowplowanalytics_snowplow_cli...` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `session_stitching` | Determines whether to apply the user mapping to the sessions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `session_timestamp` | Determines which timestamp to use for optimizing the queries in our sessionization logic. This should be set to the same timestamp as the field you partition your events table on. This does not affect how sessions are processed and all events will be processed as described in our [sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing) logic. | `collector_tstamp` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_mobile_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `enable_geolocation_context` | Flag to include the Geolocation context (device latitude, longitude, bearing, etc.) columns in the models. | `false` |
| `enable_app_errors_module` | Flag to enable the app errors module (details relating to app errors that occur during sessions). | `false` |
| `enable_application_context` | Flag to include the Application context (app version and build) columns in the models. | `false` |
| `enable_mobile_context` | Flag to include the Mobile context (device type, OS, etc.) columns in the models. | `false` |
| `enable_screen_context` | Flag to include the Screen context (screen details associated with mobile event) columns in the models. | `false` |
| `has_log_enabled` | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to `false`. | `true` |
| `platform` | A list of `platform`s to filter the events table on for processing within the package. | `['mob']` |
| `screen_view_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `screen_view` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first `page_view` or `page_ping` event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_first_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_last_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output.  See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information.Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Web data model
> Configure the legacy Web dbt package for web event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/web/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

This page helps you configure the Snowplow Web dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.1)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

> **Tip:** When modifying the `session/user_identifiers` or using `session/user_sql` in the web package these will overwrite the `domain_sessionid` and `domain_userid` fields in tables, rather than being `session/user_identifier` as in the core utils implementation. This is for historic reasons to mitigate breaking changes. Original values for these fields can be found in `original_domain_session/userid` in each table.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `heartbeat` | Page ping heartbeat time as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `10` |
| `min_visit_length` | Minimum visit length as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `5` |
| `rfc_5646_seed` | Name of the model for the RFC 5646 (language) mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_web_dim_rfc_5646_language_mapping` |
| `ga4_categories_seed` | Name of the model for the GA4 category mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_web_dim_ga4_source_categories` |
| `geo_mapping_seed` | Name of the model for the Geo mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_web_dim_geo_country_mapping` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `sessions_table` | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. `{{ ref(\'snowplow_web_sessions_custom\') }}`. Please see the [README](https://github.com/snowplow/dbt-snowplow-web/tree/main/custom_example) in the `custom_example` directory for more information on this sort of implementation. | `"{{ ref( \'snowplow_web_sessions\' ) }}"` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `list_event_counts` | A boolean whether to include a json-type (varies by warehouse) column in the sessions table with a count of events for each `event_type` in that session. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `total_all_conversions` | A boolean flag whether to calculate and add the `cv__all_volume` and `cv__all_total` columns. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/conversions/). | `false` |
| `conversion_events` | A list of dictionaries that define a conversion event for your modeling, to add the relevant columns to the sessions table. The dictionary keys are `name` (required), `condition` (required), `value`, `default_value`, and `list_events`. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/conversions/). | `""` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_stitching_id` | This is the user_id you want to stitch to sessions (and/or page views) with matching domain_userids. It supports raw `sql` expressions. | `user_id` |
| `session_stitching` | Determines whether to apply the user mapping to the sessions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `page_view_stitching` | Determines whether to apply the user mapping to the page views table. Note this can be an expensive operation to do every run. One way to mitigate this is by running this update with less frequency than your usual run by enabling this variable only for that specific run. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `false` |
| `session_timestamp` | Determines which timestamp to use for optimizing the queries in our sessionization logic. This should be set to the same timestamp as the field you partition your events table on. This does not affect how sessions are processed and all events will be processed as described in our [sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing) logic. | `collector_tstamp` |
| `cwv_days_to_measure` | The number of days to use for web vital measurements (if enabled). | `28` |
| `cwv_percentile` | The percentile that the web vitals measurements that are produced for all page views (if enabled). | `75` |
| `limit_page_views_to_session` | A boolean whether to ensure page view aggregations are limited to pings in the same session as the `page_view` event, to ensure deterministic behavior. If false you may get different results for the same `page_view` depending on which sessions are included in a run. See the [stray page ping](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/#stray-page-pings) section for more information. | `true` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_web_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `ua_bot_filter` | Flag to filter out bots via the `useragent` string pattern match. | `true` |
| `enable_iab` | Flag to include the [IAB enrichment](/docs/enriching-your-data/available-enrichments/iab-enrichment/) data in the models. | `false` |
| `enable_yauaa` | Flag to include the [YAUAA enrichment](/docs/enriching-your-data/available-enrichments/yauaa-enrichment/) data in the models. | `false` |
| `enable_ua` | Flag to include the [UA Parser enrichment](/docs/enriching-your-data/available-enrichments/ua-parser-enrichment/) data in the models. | `false` |
| `enable_consent` | Flag to enable the [consent](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/consent-module/) module. | `false` |
| `enable_cwv` | Flag to enable the [Core Web Vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/core-web-vitals-module/) module. | `false` |
| `has_log_enabled` | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to `false`. | `true` |
| `page_view_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `page_view` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first `page_view` or `page_ping` event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_first_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_last_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output.  See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information.Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Media Player data model
> Configure the Media Player dbt package for tracking video and audio playback metrics with customizable variables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/media-player/

This page helps you configure the Snowplow Media Player dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.1)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `percent_progress_boundaries` | The list of percent progress values. It needs to be aligned with the values being tracked by the tracker. It is worth noting that the more these percent progress boundaries are being tracked the more accurate the play time calculations become. Please note that tracking 100% is unnecessary as there is a separate `ended` event which the model equates to achieving 100% and it also gets included automatically to this list, in case it is not added (you can refer to the helper macro `get_percentage_boundaries` ([source](https://snowplow.github.io/dbt-snowplow-media-player/#!/macro/macro.snowplow_media_player.get_percentage_boundaries)) for details). | `[10, 25, 50, 75]` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `valid_play_sec` | The minimum number of seconds that a media play needs to last to consider that interaction a valid play. The default is 30 seconds (based on the YouTube standard) but it can be modified here, if needed. | `30` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema": "contexts_com_snowplowanalytics_snowplow_medi...` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid", "prefix...` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |
| `session_timestamp` | Determines which timestamp to use for optimizing the queries in our sessionization logic. This should be set to the same timestamp as the field you partition your events table on. This does not affect how sessions are processed and all events will be processed as described in our [sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing) logic. | `collector_tstamp` |
| `complete_play_rate` | The rate to set what percentage of a media needs to be played in order to consider that complete. 0.99 (=99%) is set as a default value here but it may be increased to 1 (or decreased) depending on the use case. | `0.99` |
| `max_media_pv_window` | The number of hours that needs to pass before new page_view level media player metrics from the `snowplow_media_player_base` table are safe to be processed by the model downstream in the `snowplow_media_player_media_stats` table. Please note that even if new events are added later on ( e.g. new `percentprogress` events are fired indicating potential replay) and the `snowplow_media_player_base` table is changed, the model will not update them in the media_stats table, therefore it is safer to set as big of a number as still convenient for analysis and reporting. | `10` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_media_player_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |
| `surrogate_key_treat_nulls_as_empty_strings` | Passed through to `dbt_utils` to match legacy surrogate key behavior. | `true` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `enable_web_events` | Whether to use the web contexts for web media events in the processing (based on the web page context). | `true` |
| `enable_mobile_events` | Whether to use the mobile contexts for mobile media events in the processing (based on the client session and screen view context). | `false` |
| `enable_ad_quartile_event` | Set to `true` if [ad quartile events](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_quartile_event/jsonschema/1-0-0) are tracked during media ad playback. | `false` |
| `enable_media_ad` | Set to `true` if the [media ad context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers along with ad events. | `false` |
| `enable_media_ad_break` | Set to `true` if the [media ad-break context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers when ad breaks are tracked along with ad events. | `false` |
| `enable_media_player_v1` | Set to `true` if the [version 1 of the media player context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0) is enabled. This schema was used in our older media plugins on the JavaScript tracker. It is not tracked in the latest versions. | `false` |
| `enable_media_player_v2` | Set to `true` if the [version 2 of the media player context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers. | `true` |
| `enable_media_session` | Set to `true` if the [media session context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers (optional but enabled by default). | `false` |
| `enable_whatwg_media` | Set to `true` if the [HTML5 media element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/media_element/jsonschema/1-0-0) is enabled. This variable is used to handle syntax depending on whether the context fields are available in the database or not. | `false` |
| `enable_whatwg_video` | Set to `true` if the [HTML5 video element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/video_element/jsonschema/1-0-0) is enabled. This variable is used to handle syntax depending on whether the context fields are available in the database or not. | `false` |
| `enable_youtube` | Set to `true` if the [YouTube context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.youtube/youtube/jsonschema/1-0-0) is enabled. This variable is used to handle syntax depending on whether the context fields are available in the database or not. | `false` |
| `ad_views_passthroughs` | Field(s) to carry through from the events table to the `snowplow_media_player_media_ad_views` derived table. The field is from the ad view event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `base_passthroughs` | Field(s) to carry through from the events table to the `snowplow_media_player_base` and `snowplow_media_player_plays_by_page_view` derived tables. The field is from the media base event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `media_event_names` | A list of names for media events to include for processing. | `['media_player_event']` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Normalize data model
> Configure the Normalize dbt package for flattening and unnesting self-describing events and entities into dedicated tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/normalize/

This page helps you configure the Snowplow Normalize dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |
| `events` | This is used internally by the packages to reference your events table based on other variable values and should not be changed. | `events` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `session_timestamp` | This determines which timestamp is used to process sessions of data. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `partition_tstamp` | This determines which timestamp is used to partition the derived tables. You need to make sure that this timestamp will be present in the flattened events table. | `collector_tstamp` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |


---

# Configure the Unified Digital data model
> Configure the Unified Digital dbt package with variables for sessions, users, and views using the interactive config generator.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/unified/

This page helps you configure the Snowplow Unified Digital dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

> **Tip:** When modifying the `session/user_identifiers` or using `session/user_sql` in the unified package these will overwrite the `domain_sessionid` and `domain_userid` fields in tables, rather than being `session/user_identifier` as in the core utils implementation. This is for historic reasons to mitigate breaking changes. Original values for these fields can be found in `original_domain_session/userid` in each table.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `enable_web` | Flag to process web events throughout the package. | `true` |
| `enable_mobile` | Flag to process mobile events throughout the package. | `true` |
| `heartbeat` | Page ping heartbeat time as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `10` |
| `min_visit_length` | Minimum visit length as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `5` |
| `rfc_5646_seed` | Name of the model for the RFC 5646 (language) mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_unified_dim_rfc_5646_language_mapping` |
| `ga4_categories_seed` | Name of the model for the GA4 category mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_unified_dim_ga4_source_categories` |
| `geo_mapping_seed` | Name of the model for the Geo mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_unified_dim_geo_country_mapping` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |
| `sessions_table` | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. `{{ ref(\'snowplow_unified_sessions_custom\') }}`. Please see the [README](https://github.com/snowplow/dbt-snowplow-unified/tree/main/custom_example) in the `custom_example` directory for more information on this sort of implementation. | `"{{ ref( \'snowplow_unified_sessions\' ) }}"` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `list_event_counts` | A boolean whether to include a json-type (varies by warehouse) column in the sessions table with a count of events for each `event_type` in that session. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `total_all_conversions` | A boolean flag whether to calculate and add the `cv__all_volume` and `cv__all_total` columns. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/). | `false` |
| `conversion_events` | A list of dictionaries that define a conversion event for your modeling, to add the relevant columns to the sessions table. The dictionary keys are `name` (required), `condition` (required), `value`, `default_value`, and `list_events`. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/). | `""` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_stitching_id` | This is the user_id you want to stitch to sessions (and/or page views) with matching domain_userids. It supports raw `sql` expressions. | `user_id` |
| `session_stitching` | Determines whether to apply the user mapping to the sessions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `view_stitching` | Determines whether to apply the user mapping to the views table. Note this can be an expensive operation to do every run. One way to mitigate this is by running this update with less frequency than your usual run by enabling this variable only for that specific run. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `false` |
| `session_timestamp` | Determines which timestamp to use for optimizing the queries in our sessionization logic. This should be set to the same timestamp as the field you partition your events table on. This does not affect how sessions are processed and all events will be processed as described in our [sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing) logic. | `collector_tstamp` |
| `conversion_stitching` | Determines whether to apply the user mapping to the conversions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `custom_sql` | This allows you to introduce custom sql to the `snowplow_unified_events_this_run` table, which you can then leverage in downstream models. For more information on the usage, see the following page on the [advanced usage of the utils package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/additional-sql-on-events-this-run/). | `""` |
| `cwv_days_to_measure` | The number of days to use for web vital measurements (if enabled). | `28` |
| `cwv_percentile` | The percentile that the web vitals measurements that are produced for all page views (if enabled). | `75` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_unified_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |
| `use_refr_if_mkt_null` | Use the refr fields when the mkt fields are null for the definition of default channel group. This is a common case when for example a landing page has a redirect. | `false` |
| `enable_initial_checks` | Use this field to test the existance of all the seed models in the dwh and the existence of contexts that are needed based on your configuration. This is useful when changing the package configuration to ensure quick fails if the configuration is incorrect. | `false` |
| `allow_null_dvce_tstamps` | Use this field to bypass optimization which avoids long scans due to late arriving data. Warning, use this field only when using specific older versions of the AMP tracker or the Pixel tracker. | `false` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `ua_bot_filter` | Flag to filter out bots via the `useragent` string pattern match. | `true` |
| `enable_conversions` | Flag to enable the conversions optional module. | `false` |
| `enable_deep_link_context` | Flag to include the deep link context data in the models. | `false` |
| `enable_geolocation_context` | Flag to include the geolocation data in the models. | `false` |
| `enable_iab` | Flag to include the [IAB enrichment](/docs/enriching-your-data/available-enrichments/iab-enrichment/) data in the models. | `false` |
| `enable_yauaa` | Flag to include the [YAUAA enrichment](/docs/enriching-your-data/available-enrichments/yauaa-enrichment/) data in the models. | `false` |
| `enable_ua` | Flag to include the [UA Parser enrichment](/docs/enriching-your-data/available-enrichments/ua-parser-enrichment/) data in the models. | `false` |
| `enable_browser_context` | Flag to include fields extracted from the [Browser context](/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/ootb-data/device-and-browser/index.md#browser-context) data in the models (JS tracker <3.17), can be used together with `snowplow__enable_browser_context_2` (supports 2nd major schema version) to coalesce the two. | `false` |
| `enable_browser_context_2` | Flag to include fields extracted from the 2nd major version of schemas related to the [Browser context](/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/ootb-data/device-and-browser/index.md#browser-context) data in the models (JS tracker (>=3.17), can be used together with `snowplow__enable_browser_context` for previous versions to coalesce the two. | `false` |
| `conversion_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the events_this_run table therefore taking all events. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output.  See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information.Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |
| `enable_app_errors` | Flag to include the mobile app error data in the models. | `false` |
| `enable_application_context` | Flag to include the app context data in the models. | `false` |
| `enable_consent` | Flag to enable the [consent](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/) module. | `false` |
| `enable_cwv` | Flag to enable the [Core Web Vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/) module. | `false` |
| `enable_mobile_context` | Flag to include mobile context data in the models | `false` |
| `enable_screen_context` | Flag to include the mobile screen data in the models. | `false` |
| `enable_screen_summary_context` | Flag to process the screen engagement information in the screen summary context on mobile events. | `false` |
| `has_log_enabled` | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to `false`. | `true` |
| `session_aggregations` | Aggregations to calculate as part of the the derived table. A list of dictionaries defining the type (sum, avg, min, max, count, countd), field to aggregate, and what to alias the column. | `[ ] (no aggregations)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first `page/screen_view` or `page_ping` event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_aggregations` | Aggregations to calculate as part of the the derived table. A list of dictionaries defining the type (sum, avg, min, max, count, countd), field to aggregate, and what to alias the column. | `[ ] (no aggregations)` |
| `user_first_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_last_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information. Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |
| `view_aggregations` | Aggregations to calculate as part of the the derived table. A list of dictionaries defining the type (sum, avg, min, max, count, countd), field to aggregate, and what to alias the column. | `[ ] (no aggregations)` |
| `view_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `page/screen_view` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |
| `snowflake_lakeloader` | To use the models on an iceberg events table created via the lakeloader on Snowflake, this will need to be enabled due to different table structures. | `false` | Snowflake |


---

# Configure the Utils data model
> Configure the Utils dbt package with core macros and helper functions for Snowplow data modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/

This page helps you configure the Snowplow Utils dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `events_schema` | The name of the schema that contains your atomic events. | `events` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `session_timestamp` | Determines which timestamp to use for optimizing the queries in our sessionization logic. This should be set to the same timestamp as the field you partition your events table on. This does not affect how sessions are processed and all events will be processed as described in our [sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing) logic. | `collector_tstamp` |
| `custom_sql` | This allows you to introduce custom sql to the `snowplow_base_events_this_run` table, which you can then leverage in downstream models. For more information on the usage, see the following page on the [advanced usage of the utils package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/additional-sql-on-events-this-run/). | `""` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `package_name` | The name of the package you are running this macro under. This has implications for your `manifest` table. | `snowplow` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_web_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `entities_or_sdes` | A list of dictionaries defining the `context` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Adding fields to a derived table
> Add custom fields to Snowplow derived tables using passthrough fields, custom aggregations, or custom derived models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/

You may often wish to add fields onto the derived tables within our packages; perhaps there is some additional dimension you need for your analysis, or some extra calculation you wish to perform. In most cases this is relatively straight forward to do, and even in the most complex case you don't need to alter the logic for the derived table itself.

## Option 1: Passthrough fields

Where possible, if the field you want to add is attached to the original event for that table (e.g. the page/screen view for views) then you can make use of the [passthrough fields](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) feature in our packages.

## Option 2: Custom aggregations

If you require aggregation at the level of the table (e.g. session identifier for sessions) you can make use of the [custom aggregations](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/) feature in our packages that support it.

## Option 3: Custom derived table

If none of the above options are suitable, the best approach is to build a custom version of our derived table, and read from the existing [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#other-this-run-tables) table to then add in your additional fields.

> **Tip:** For Redshift, if you need any additional self-describing event or entity fields in the events this run table, check out the [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/#custom-entities--events) page for how to add these.

As an example, let's say we want to add some field to the sessions table in the unified package, that for simplicity is just going to be an `is_test` calculation based on the app\_id.

### Disable the derived model in the package

The first step is to disable the derived model in the package itself. You do this in your top level project yaml:

```yaml
...

models:

  snowplow_unified:

    sessions:

      snowplow_unified_sessions:

        +enabled: false
```

### Add a new derived model

Next we add a new model to replace the one we just disabled; the easiest thing to do is just copy the contents of the model we just disabled. While you could name this anything, it's easiest for downstream use cases to keep the original name

```jinja2
{{

  config(

    tags=['snowplow_unified_incremental', 'derived'],

    ...

  )

}}



select *

  {% if target.type in ['databricks', 'spark'] -%}

  , DATE(start_tstamp) as start_tstamp_date

  {%- endif %}

from {{ ref('snowplow_unified_sessions_this_run') }}

where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
```

### Modify the new model as needed

```jinja2
{{

  config(

    ...

  )

}}



select *

  , case when app_id like '%_test' then 'test' else 'prod' end as app_type

  {% if target.type in ['databricks', 'spark'] -%}

  , DATE(start_tstamp) as start_tstamp_date

  {%- endif %}

from {{ ref('snowplow_unified_sessions_this_run') }}

where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
```

> **Tip:** If you are using BigQuery, you should look at the [combine column versions](https://github.com/snowplow/dbt-snowplow-utils?tab=readme-ov-file#combine_column_versions-source) macro we provide to automatically combine minor versions of your schemas in the model.

### (Optional) Backfill the model

Follow the steps to [backfill models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/) the model if you want this field to be populated historically.

---

# Additional SQL on events this run
> Add custom SQL to the events this run table using the snowplow__custom_sql variable for calculated fields and context extraction.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/additional-sql-on-events-this-run/

There may be times when the [events this run table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) requires additional fields on it for you to make use of to [add to your derived tables](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/). It is not recommended to alter this model directly, as it is a core part of the packages, but you can use the `snowplow__custom_sql` variable in packages that support it to add custom sql into the `select` block of the events this run model.

To find out if your package supports this, check the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) page.

## Using `snowplow__custom_sql`

The `snowplow__custom_sql` variable is passed as-is through to the `select` block of the [`events_this_run` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/19bfd655fea1338f28cd6b2f8ca5863cc137aac7/macros/base/base_create_snowplow_events_this_run.sql#L39). Because of where this is passed in, fields such as the session and user identifiers are not available at this point, but any other field should be. In the case of Redshift any self-describing events or entities provided in the [`snowplow__entities_or_sdes` variable](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) will be available with the provided prefix and table alias.

Say you wanted to add a calculated field based on your app id, you would set this as such:

```yml
...

vars:

  snowplow_<package_name>:

    snowplow__custom_sql: "case when app_id like '%_test' then 'test' else 'prod' end as app_type"

...
```

### Utilizing custom contexts or SDEs

Suppose you have a custom context called `contexts_com_mycompany_click_1` which contains a `click_id` that you want to concat with Snowplow's `domain_sessionid`. If you want to make this available in the events this run table directly you could do the following:

**Databricks & Snowflake:**

```yml
vars:

    ...

    snowplow__custom_sql: "CONCAT(com_mycompany_click_1[0].click_id, '_', domain_sessionid) as click_session_id"

    ...

...
```

**BigQuery:**

```yml
vars:

    ...

    snowplow__custom_sql: "CONCAT(com_mycompany_click_1[safe_offset(0)].click_id, '_', domain_sessionid) as click_session_id"

    ...

...
```

**Redshift & Postgres:**

```yml
vars:

    ...

    snowplow__custom_sql: "com_mycompany_click_1.click_id || '_' || domain_sessionid as click_session_id"

    ...

...
```

***

### Adding multiple fields

If you'd like to add multiple lines of SQL, you can do that as well by making this string a multi-line string, ending each non-final line with a comma. You can do that as follows:

```yml
vars:

    ...

    snowplow__custom_sql: |

        com_mycompany_click_1.click_id || '_' || domain_sessionid as click_session_id,

        case when app_id like '%_test' then 'test' else 'prod' end as app_type

    ...

...
```

---

# Daily aggregate model
> Build daily aggregate tables in Snowplow dbt packages by scanning derived tables with date range filters for accurate aggregations.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/daily-aggregate-table/

Because of the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of our packages, all of the [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/) tables contain events (or a higher level) for only the sessions being processed in that run. For this reason, it is not possible to build daily aggregate tables directly off of any this run tables.

To better understand why, consider a late arriving event for a session that started 2 days ago; because we would reprocess all events for this session our sessions this run table would contain a record for that session with a start date 2 days ago, but all other sessions for that day would already have been processed and would **not** be in this table. If you try to get a count of sessions, you'll report that there was only 1 session that day. You could try to be clever with incremental additions, but this session may have already been partially processed so may or may not have already been counted.

The only way to accurately do daily aggregates in our packages is to calculate these off the _derived tables_, i.e. the persistent tables in the package. This would be inefficient to scan the whole table every time, so we can be smart and use the _range of dates_ in the [events this run table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) to identify which dates we need to rescan, thus reducing the compute cost.

## A simple example

Let's say we want to do a simple daily aggregates model built in the Unified package - we want to keep a count of the number of sessions and the average engaged time per day. This table would be 1 row per day, and we would need to read from the `snowplow_unified_sessions` _derived_ table for the reasons discussed above.

The skeleton of the model would be to make an incremental model, ensure we add the [optimized upsert](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/), checking for new events, and selecting from the sessions model.

Note there is no need to [tag](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/#tagging-models) the model as this is not reading directly from a this run table.

```jinja2
{{

  config(

    materialized='incremental',

    unique_key='session_date',

    upsert_date_key='session_date',

    tags=['derived'],

    snowplow_optimize = true

  )

}}



-- depends_on: {{ ref('snowplow_unified_events_this_run') }}



{# Get the range of dates processed in the current run to limit the later scan #}

{% if is_incremental() %}

{%- set lower_limit, upper_limit = snowplow_utils.return_limits_from_model(ref('snowplow_unified_events_this_run'),

                                                                           'derived_tstamp',

                                                                           'derived_tstamp') %}

{% endif %}



select

  date(start_tstamp) as session_date,

  count(*) as num_sessions,

  avg(engaged_time_in_s) as average_engaged_time

from {{ ref('snowplow_unified_sessions') }}

where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.

{% if is_incremental() %}

  and start_tstamp >= date({{ lower_limit }})

  and start_tstamp <= {{ dateadd(datepart="day", interval=1, from_date_or_timestamp= upper_limit) }}

{% endif %}

group by 1
```

> **Tip:** Notice that we are filtering on the `start_tstamp` column from the sessions table, because this is the partition/sort key for that table. Make sure you used the right column for the derived table you are querying.

We need to add a dependency on events this run to ensure dbt runs everything in the right order, we do this by adding the `-- depends_on: {{ ref('snowplow_unified_events_this_run') }}` comment. Next we get the limits for the current run and those are the dates we then use to work out what range we have to scan the sessions table on. On the first run, it will do a full scan to backfill the table from the existing derived sessions data.

You could add additional buffers around the limits if required, but make sure that you are always processing full days of data from the sessions table (at least up to the current max timestamp). You can also extend this as needed to add other fields, joins, or anything else you require.

If you need weekly/monthly aggregates, the ideas are the same, just make sure you are always scanning and reprocessing a whole period in each run.

---

# Incorporating non-Snowplow data
> Incorporate external data sources into Snowplow derived tables by joining non-Snowplow data with custom derived models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/incorporating-non-snowplow-data/

There may be times when you wish to add some fields to our derived tables that are not coming from the events table itself but from your own external data sources.

Our packages are designed to process events associated to sessions, based on newly incoming events from the `atomic.events` table. Allowing for incorporating external data into the processing by default is not planned so far, not only because there can be a myriad of ways how different users would want to link their data to our events/sessions, but also because the package would then need some sort of governance over the data source being readily available for the period to be processed, which it has no impact over. However, there is a way to accomplish that through custom models.

**Example Scenario 1:** Imagine you run an ecommerce site, and you are only tracking the products\` SKU, but you have an internal table with richer information about the particular product. You would like to add some extra information to your views table through the custom page\_view context.

**Example Scenario 2:** Wanting to incorporate additional user based information into the sessions table with the use of the external User Profiles table.

In both cases what you could do is follow the recommendations on [adding a custom derived table](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/#option-3-custom-derived-table), basically by disabling the package provided derived table, copying the content of the existing derived table as is, which is normally just a select \* from the upstream this\_run table.

> **Info:** You might want to add extra safety measures to avoid misaligned data between the package and the table you want to bring data from. Consider using a pre-hook that you can add to the config section of your derived table for instance. Update schedules would also need to be synced for a hassle-free run and consider the impact when there are changes, refreshes to the source table, in case you need to run a full or partial refresh afterwards.

When modifying the model you would typically want to add left joins using the key that makes sense for your use case (e.g. user\_identifier, session\_identifier, view\_id etc.) and list the additional field(s) which you would like to be incorporated. As long as you keep a `this_run` table as a basis and select everything from it, and also tag your custom model as `snowplow_{{package_name}}_incremental` it will become part of the run and the package will keep everything in sync through the manifest table.

Bear in mind that if you make changes to the Sessions table, you might break the Users table in the Unified Digital package depending on the changes as the Users table is built on top of the Sessions table (e.g. it is generally safe to add more fields (be careful about name clashes), though you might also need to add them as a [passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) for them to show up in the Users table, but probably not to remove them, best to test it first), otherwise they are safe to update independently.

> **Info:** You can inspect the lineage graph to verify this if you are unsure by using dbt's built in data lineage feature through dbt docs. We also update that for each of our latest releases on github: <https://snowplow.github.io/dbt-snowplow-unified/#!/overview?g_v=1>

---

# Example custom dbt models
> Example custom models for Snowplow dbt packages including derived tables, aggregations, and sessionization patterns.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/

The following pages contain specific examples for building custom models to do specific things. They are meant to help you understand how to go about building certain types of custom models and are by no means an exhaustive list of what you can do, and you can mix and match elements to fit your needs.

If you would like to see an example of how this implementation works in practice, check out our [example dbt project](https://github.com/snowplow-incubator/dbt-example-project/tree/main/custom_event_table_unified) on Github. It demonstrates how the Unified Digital package can be used as a base for customizations, allowing you modify its behavior without forking the package (e.g. macro overwrite, adding custom data models, overwriting variables etc.).

---

# New derived model
> Create new incremental derived models in Snowplow dbt packages building from events this run with optimized upserts.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/new-derived-model/

We have always seen our packages as a starting point for your analysis and insight generation; there to get you started and give you an idea of what you can do, while providing the framework to build your own models on top to meet your bespoke needs. Assuming that customizing the existing derived tables does not suit your needs the next steps is to create a custom derived model.

As you are building this for yourself, there is no need to follow our usual this run -> derived approach, and you can just build an incremental model on top of events this run!

> **Note:** Any custom derived model must build only to session-level or _lower_ because of the [incremental sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of our packages. For example you can't build a table that aggregates per day using the standard sessionisation as not all events from a given day are always processed in the same run, at least not on any of the [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/), see the [Daily Aggregate Table](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/daily-aggregate-table/) for how to do this.

## A simple example

Let's say we want to do a very simple model built in the unified package - we want to keep a count of the number of each type of event per session in a table. This table would be 1 row per session+event name, and can build incrementally as it's at a lower level than session.

The skeleton of the model would be to make an incremental model, ensure we add the [optimized upsert](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/), [tag](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/#tagging-models) the model, checking for new events, and selecting from [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run).

```jinja2
{{

  config(

    materialized='incremental',

    unique_key='session_identifier',

    upsert_date_key='start_tstamp',

    tags=['snowplow_unified_incremental', 'derived'],

    snowplow_optimize = true

  )

}}



select *

    ...

from {{ ref('snowplow_unified_events_this_run') }}

where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
```

Then it is simply a case of writing the actual model content itself, which in this case is very straight forward. You may have a more complex model, perhaps bringing in data from another source or model in your dbt project, but the idea is the same. Note that we need a date key to be able to use our optimized upsert.

```jinja2
{{

  config(

    materialized='incremental',

    unique_key='session_identifier',

    upsert_date_key='start_tstamp',

    snowplow_optimize = true

  )

}}



select

    session_identifier,

    min(derived_tstamp) as start_tstamp,

    event_name,

    count(*) as event_vol

from {{ ref('snowplow_unified_events_this_run') }}

where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.

group by 1, 3
```

Follow the steps to [backfill](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/) the model. Make sure to add any additional config for optimizing your tables based on your warehouse.

---

# Using different sessionisation
> Use custom session identifiers in Snowplow dbt packages including daily aggregates and alternative sessionization logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-different-sessionisation/

Using a different sessionisation is a rare use case for our packages, but can be done in 2 different ways. The use case for each approach is often different so we treat them differently on this page.

## Tweaking the session identifiers

If you only need to slightly alter the identifiers, such as using an entity value where it exists instead of just the default fields, you can do this by following the details in the [custom identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) page to alter this via the existing variables of our packages.

## Using the incremental logic for all custom models

The more large-scale use case is when you wish to take advantage of our [incremental sessionisation logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), but have no need for any of the other derived tables in our packages. A common use case for this is when you want to do daily aggregates and reporting, in this case your "session" identifier is actually the day of the event.

For this purpose, rather than disable all the models you don't need, it would be easier to use the `base` macros and tables at the root of your project, or stand up a standalone project. For this purpose we have the [dbt-template](https://github.com/snowplow-incubator/dbt-template) repository which has all the core logic of our packages set up, including all the variables and models.

Once you have forked/copied this project, you will need to replace all instances of `<YOUR_PACKAGE_NAME>` across the package with the name you want for this project. This includes file names, not just a find and replace in the files.

You can then make changes to the variables as required, for example the following changes would give you a daily "sessionisation" to allow you to easily do incremental daily reporting:

```diff
...

vars:

- <YOUR PACKAGE NAME>:

+ my_package:

    ...

    snowplow__app_id: []

-   # snowplow__session_sql: 'e.domain_sessionid'

+   snowplow__session_sql: "DATE(e.derived_tstamp)"

    # snowplow__user_sql: 'e.domain_userid'

    # snowplow__custom_sql: ''

    ...

...
```

---

# Using multi-valued entities
> Extract and use multi-valued entities in Snowplow dbt custom models with warehouse-specific unnesting techniques.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/

When building [custom derived models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/new-derived-model/) you may find yourself needing to use a multi-valued entity, i.e. an entity with multiple entries in the array. These fields should not be extracted as part of the [event this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table because it would lead to duplicate rows, and so are only extracted as part of the custom model where you are either keeping the rows at that level or aggregating back up.

In this page we will just showcase the SQL required, including some of our specialist macros, to extract out these entities into a 1-row per value table, for you to then use in any specific model.

## SQL to extract each value

**Snowflake:**

To make use of a multi-valued entity in BigQuery, you can make use of the `LATERAL FLATTEN` function.

```jinja2
SELECT

    ...

    r.value:myField::varchar,  -- field will be variant type so important to cast

    ...

FROM

    {{ ref('snowplow_<package_name>_base_events_this_run') }} t,

    LATERAL FLATTEN(input => t.contexts_my_entity_1) r
```

**BigQuery:**

To make use of a multi-valued entity in BigQuery, you can make use of the `unnest` function. Note that you will need to do this for each version of the context.

```jinja2
SELECT

    ...

    my_ent.my_field AS my_field,

    ...

FROM

    {{ ref('snowplow_<package_name>_base_events_this_run') }}

LEFT JOIN

    unnest(contexts_my_entity_1_0_0) AS my_ent -- left join to avoid discarding events without values in this entity
```

**Databricks:**

To make use of a multi-valued entity in BigQuery, you can make use of the `LATERAL VIEW EXPLODE` function.

```jinja2
SELECT

    ...

    my_ent.my_field,

    ...

FROM

    {{ ref('snowplow_<package_name>_base_events_this_run') }}

    LATERAL VIEW EXPLODE(contexts_my_entity_1) AS my_ent
```

**Redshift:**

As the fields in Redshift are part of a different table, we need to join that table onto the events this run model. We do this in 3 steps:

1. Get the limits for the current run to reduce the scan on the entity table
2. Use a CTE and macro to get the records from the entity table, adding fields needed to deduplicate
3. Join this onto events this run, using a specific set of conditions to manage duplicates but keep _genuine_ same-valued records.

We use the [`snowplow_utils.get_sde_or_context` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/utils/get_sde_or_context.sql), which takes five arguments:

- `schema`: the schema in your warehouse that the table is in
- `identifier`: the table name for the entity
- `lower_limit`: the lower limit of the event time to scan
- `upper_limit`: the upper limit of the event time to scan
- `prefix`: the prefix to apply to all column names
- `single_entity`: if it is a single-valued entity

In this case, we want to make sure to set `single_entity` to false as we are explicitly using a multi-valued entity.

```jinja2
{# Get the limits for the run to reduce the scan on the entity table#}

{%- set lower_limit, upper_limit = snowplow_utils.return_limits_from_model(ref('snowplow_<package_name>_base_sessions_this_run'),

                                                                            'start_tstamp',

                                                                            'end_tstamp') %}



{# Use the macro as part of a CTE which will select the relevant data, prefix the columns, and add the de-dupe fields #}

with {{ snowplow_utils.get_sde_or_context(var('snowplow__atomic_schema', 'atomic'), 'contexts_my_entity_1', lower_limit, upper_limit, 'my_ent', single_entity = false) }},



select

    a.*,

    b.my_ent_my_field

from {{ ref('snowplow_<package_name>_base_events_this_run') }} a

left join contexts_my_entity_1 b on

    a.event_id = b.yauaa__id  -- match the event id between the tables, note the dobule underscore

    and a.collector_tstamp = b.yauaa__tstamp -- match the collector timestamp between the tables, note the double underscore

    and mod(b.yauaa__index, a.event_id_dedupe_count) = 0 -- ensure one version of each potentially duplicated entity in context
```

***

This table can then be used to do any further analysis required, either in the same model or for use in another model.

---

# High volume optimizations
> Optimize Snowplow dbt packages for high-volume data processing with reduced columns, ephemeral models, and tuned incremental logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/

For users with very high data volumes (>100M daily events) you may find that, even with our [optimized upserts](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) and [incremental sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), the package is still slow in any given run or the processing cost is high. There are a few specific things you can do to help optimize the package even further, which require various levels of effort on your part. In general we have taken the decision to not do these things as part of the "normal" deployment of our packages as there is a trade-off for each one and in the vast majority of use cases they aren't required.

## Tune the incremental logic parameters

Our [incremental sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) by default will look back and reprocess 6 hours of data; if you are confident there is no delay in data loading, or if you are using the `load_tstamp` as the `snowplow__session_timestamp` then there is much less need to have such a big lookback window, decreasing this to 1 hour will greatly reduce volumes of data for regular (\~hourly) package runs.

Decreasing the backfill limit days will only impact on backfill runs so once models are up to date this will have little impact. Decreasing the upsert lookback days, or the session lookback days can have benefits but come at the risk of duplicates making it into the manifest or derived tables so do this at your own risk.

## Reduce the number of columns written + ephemeral this run

> **Note:** These configurations have not been fully tested and leave you at risk of issues occurring when we make upgrades to our packages. Continue at your own risk.

### Derived Tables

In general we try to only write the columns into [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/), and therefore derived tables, that are populated and relevant to that table. However, there may by many columns in these tables that you don't require and as as cloud warehouses are usually [columnar warehouses](https://en.wikipedia.org/wiki/Column-oriented_DBMS) this leads to more compute required to write these columns, and more storage to keep and maintain the metadata for them.

Rather than edit the this run tables directly, as this could leave you out of sync with our updates, we recommend changing the materialization of all this run models (except the events and base sessions) to be `ephemeral`, this stops the incremental data being first written as a table before being merged into the derived table. Doing this may not be the best option if you re-use any of these tables in multiple custom models as it will have to run them each time.

Next instead of selecting all columns from your now ephemeral this run models, disable our derived model and build your own to explicitly select just the columns that you require. See the [adding fields to derived table](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/) example for how to do this.

Your derived table will now be writing and storing fewer columns, and the this run tables will optionally not be written at all.

### Events this run

The [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) by design contains every column that your atomic events table does including all self-describing events and entities (except in Redshift where these must be specified by a variable to be joined on). This can mean a very large amount of columns and data that you never use have to be read and written in every single run.

In general, it is a bad idea to alter the events this run table, as it can lead to issues elsewhere in the package, but it is also one of the largest places to reduce the amount of read/written data in a given run. The easiest thing to do in this case is to disable the events this run table from the package and duplicate the model into your top level project. Note the model must have the exact same name as the one you disabled, and you will need to duplicate many of the variables from the package into your project yaml to ensure they are available in the correct scope.

Nearly all our packages will have the same structure for this model; a macro is called, this is used within a CTE, and then we do some further SQL on this CTE. In your case, you could select only the columns required for **all** downstream models explicitly within this model

#### Example from unified

**Project yaml**

Note that any variables used in the model need to be made available in your project scope

```yaml
models:

  snowplow_unified:

    base:

      scratch:

        snowplow_unified_base_events_this_run:

          +enabled: false



vars:

  my_project_name:

    snowplow__session_timestamp: ...

    ...
```

**snowplow\_unified\_base\_events\_this\_run (copy)**

We have removed the vast majority of the code from the original model code. The changes are the addition of the `default_cte` and then the selection of specific columns below.

```jinja2
...

with base_query as (

  {{ base_events_query }}

)



default_cte as (

  select

    *

    -- extract commonly used contexts / sdes (prefixed)

    {{ snowplow_unified.get_web_page_context_fields() }}

    {{ snowplow_unified.get_iab_context_fields() }}

    {{ snowplow_unified.get_ua_context_fields() }}

    {{ snowplow_unified.get_yauaa_context_fields() }}

    {{ snowplow_unified.get_browser_context_fields() }}

    {{ snowplow_unified.get_screen_view_event_fields() }}

    {{ snowplow_unified.get_session_context_fields() }}

    {{ snowplow_unified.get_mobile_context_fields() }}

    {{ snowplow_unified.get_geo_context_fields() }}

    {{ snowplow_unified.get_app_context_fields() }}

    {{ snowplow_unified.get_screen_context_fields() }}

    {{ snowplow_unified.get_deep_link_context_fields() }}

    {{ snowplow_unified.get_app_error_event_fields() }}

    {{ snowplow_unified.get_screen_summary_context_fields() }}





  {% if var('snowplow__enable_consent', false) -%}

    {{ snowplow_unified.get_consent_event_fields() }}

    {{ snowplow_unified.get_cmp_visible_event_fields() }}

  {% endif -%}



  {% if var('snowplow__enable_cwv', false) -%}

    {{ snowplow_unified.get_cwv_fields() }}

  {% endif -%}

  from base_query

)



select

  ... -- your specific columns here

from

  default_cte
```

> **Tip:** In the case of the unified package, there are two event this run tables, normally we recommend editing the latter one via the macro that creates it, but in this case you need to edit the upstream one to remove the columns as early as possible.

---

# Custom dbt models
> Create, update, and remove custom models in Snowplow dbt packages with incremental, drop+recompute, and this run model types.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/

The Snowplow packages are designed to be easily customized or extended within your own dbt project by building your own custom models. The standard models we provide (base, page/screen views, sessions and users) are not designed to be modified by you.

In general there are 3 types of custom models you may end up building:

1. **Drop+recompute/view models**: these models build from the derived tables but require a Drop+recompute each time they are built. This could include things like daily aggregations or rolling views.
2. **Incremental models**: these models build from the existing (or custom) [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#other-this-run-tables) tables(s) and are incrementally upserted to each run. This could include things like a custom sessions table with additional user information. These are the most common types of custom models.
3. **This run models**: these models build from the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table that contains all columns for events being (re)processed in the run. These can be used to feed custom incremental models.

Alternatively, if you wish to only use the events this run logic and build everything on top of this yourself, you may be better looking at the [using different sessionisation example](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-different-sessionisation/) for how to set up a new package from scratch.

The best way to learn is by doing, so check out the [Examples](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/) section for specific custom models.

## Guidelines & Best Practice

The standard models carry out the heavy lifting in establishing an incremental structure and providing the core logic for the most common aggregation use cases. It also allows custom models to be plugged in without impeding the maintenance of standard models.

The following best practices should be followed to ensure that updates and bug fixes to the model can be rolled out with minimal complication:

- Custom models should not modify any of the tables generated by the Snowplow packages i.e. the scratch, derived or manifest tables.
- The logic for custom SQL should be idempotent, and restart-safe - i.e. it should be written in such a way that a failure mid-way, or a re-run of the model will not change the deterministic output.
- Redshift/Postgres users: be careful about joining mutli-valued entities to any events table. Deduplication logic needs to be applied to avoid many-to-many joins. See the [using multi-valued entities example](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/) for how to manage this.

In short, the standard models can be treated as the source code for a distinct piece of software, and custom models can be treated as self-maintained, additive plugins - in much the same way as a Java package may permit one to leverage public classes in their own API, and provide an entry point for custom programs to run, but will not permit one to modify the original API.

The [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/) and derived tables are considered part of the 'public' class of tables in this model structure, and so we can give assurances that non-breaking releases won't alter them. The other tables may be used in custom SQL, but their logic and structure may change from release to release, or they may be removed.

## Custom model usage

An overview of the detail provided below is available in this summary table:

| Model Type           | Materialization             | Requires Tagging | Requires backfilling | Requires specific refreshing | Use is run with new events | Use `snowplow_optimize` |
| -------------------- | --------------------------- | ---------------- | -------------------- | ---------------------------- | -------------------------- | ----------------------- |
| Drop+recompute/ view | `table`/`view`              | ❌                | ❌                    | ❌                            | ❌                          | ❌                       |
| Incremental          | `incremental`               | ✅                | ✅                    | ✅                            | ✅                          | ✅                       |
| This Run             | `table`/`view`/ `ephemeral` | ✅                | ❌                    | ❌                            | ❌                          | ❌                       |

### Tagging Models

Any incremental model, or one that builds from the events this run table, in the package requires tagging so that it can be tracked in the manifest table as part of the [incremental sessionisation logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/). If you do not correctly tag the model you may get an error, or your package may get stuck in State 2.

The model(s) should be tagged with `snowplow_<package>_incremental`; the easiest way to achieve this is to place any custom models of this type in a specific directory in your project and tag the whole folder in your project yaml:

```yml
models:

  your_dbt_project:

    snowplow_<package>_custom_models:

      +tags: snowplow_<package>_incremental #Adds tag to all models in the 'snowplow_<package>_custom_models' directory
```

Note that because Drop+recompute models are not tagged, they will not be run when using the `snowplow_<package>` selector.

### Using `is_run_with_new_events`

Incremental type custom models, those that reference a `_this_run` type table should make use of the `is_run_with_new_events` macro to only process the table when new events are available in the current run.

```jinja2
select

  ...

from {{ ref('snowplow_<package>_<table>_this_run') }}

where {{ snowplow_utils.is_run_with_new_events('snowplow_<package>') }} --returns false if run doesn't contain new events.
```

### Using `snowplow_optimize`

To make use of our [efficient upsert](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) you should add `snowplow_optimize=true` to your custom model config for incremental models, setting an appropriate `upsert_date_key` and `unique_id`.

```jinja2
{{

  config(

    materialized='incremental',

    upsert_date_key='my_date_field',

    unique_key='unique_id_field',

    snowplow_optimize=true,

    ...

  )

}}



select

  ...
```

### Backfilling Custom Models

See our page on [backfilling](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/).

### Refreshing Custom Models

See our page on [full or partial refreshes](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/).

### Retiring Custom Models

If you want to retire a custom model, you should delete the models from your project or [disable](https://docs.getdbt.com/reference/resource-configs/enabled#disable-a-model-in-a-package-in-order-to-use-your-own-version-of-the-model) the models.

There is no need to remove the models from the `snowplow_<package>_incremental_manifest` manifest table. The packages identifies **enabled** models tagged with `snowplow_<package>_incremental` within your project and selects these models from the manifest in order to calculate the state of the package as described above.

> **Danger:** Do **NOT** just use `--exclude` on the retired models from your job in production. Currently the packages is unable to identify which models are due to be executed in a given run. As a result, if you exclude a model the package will get stuck in [State 3](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-3-models-out-of-sync) and continue to attempt to sync your excluded with the remaining models.

## Tips for developing custom models

> **Note:** Most of these tips apply only to Incremental type custom models

While developing custom models you may benefit from the following:

1. Minimizing the amount of data being processed to reduce cost & run time.
2. Use recent events from your events table to ensure you have all the latest contexts and event types available.
3. BigQuery only: Automatic handling of evolving schemas for custom contexts and unstructured events.

### 1. Reducing Costs

By setting `snowplow__backfill_limit_days` to 1 in your `dbt_project.yml` file you will only process a days worth of data per run.

We have provided the `get_value_by_target` macro to dynamically switch the backfill limit depending on your environment i.e. dev vs. prod, with your environment determined by your target name:

```yml
vars:

  snowplow_<package>:

    snowplow__backfill_limit_days: "{{ snowplow_utils.get_value_by_target(

                                            dev_value=1,

                                            default_value=30,

                                            dev_target_name='dev') }}"
```

### 2. Using Recent Data

This can be achieved by setting `snowplow__start_date` to a recent date. If a custom model only models newly tracked events, you should set this to the date that data started to be produced instead of your initial pipeline date. To dynamically change the start date depending on your environment, you can use the following:

```yml
vars:

  snowplow_<package>:

    snowplow__start_date: "{{ snowplow_utils.get_value_by_target(

                                      dev_value=snowplow_utils.n_timedeltas_ago(1, 'weeks'),

                                      default_value='2020-01-01',

                                      dev_target_name='dev') }}"
```

### 3. Handling of schema evolution

> **Info:** As your schemas for such custom contexts and unstructured events evolve, multiple versions of the same column will be created in your events table e.g. `custom_context_1_0_0`, `custom_context_1_0_1`. These columns contain nested fields i.e. are of a datatype `RECORD`. When modeling Snowplow data it can be useful to combine or coalesce each nested field across all versions of the column for a continuous view over time.
>
> The snowplow-utils package provides the [combine\_column\_versions](https://github.com/snowplow/dbt-snowplow-utils#combine_column_versions-source) macro, which will automatically coalesce the fields within every version of the specified column. This mitigates the need for you to update your models every time a new column version is created.
>
> Please refer to the [snowplow-utils](https://github.com/snowplow/dbt-snowplow-utils) docs for the full documentation on these macros.

---

# Custom Attribution implementations
> Customize the Attribution dbt package with custom conversion periods, sources, and attribution overview logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/custom-implementations/

> **Note:** Make sure you upgrade to [v.0.4.0](https://github.com/snowplow/dbt-snowplow-unified/releases/tag/0.4.0) of Unified Package if you would like to use the v0.2.0 of Attribution Package

This page describes how to customize the Attribution model.

## Customizing the Attribution Overview

The Attribution Overview, though enabled by default, is abstracted away into its own macro: `attribution_overview()` encouraging the package users to have a think about customizing the logic as it is heavily generalized by nature. Let's break down what is happening there by default so that the users can understand it better and be able to make customizations that fits their analysis best.

When analyzing marketing attribution it is always going to be a specific analysis with respect to time: there is always a conversion period, which sets the main timeframe of the analysis, during which conversions happen.

By default, the model expects a spend source and the model aggregates that (it takes the sum of total cost by spend and by channel and campaign). However, associating the spend data with the attribution data is taken dynamically by default based on the timestamp of the conversion relative to the spend. The view by default takes into account spend data that happened before conversion but no less than 90 days before conversion. This is where it makes sense add custom logic, adjusting the period to filter on the investment period relative to the conversion period in question depending on business requirements.

```sql
    from spend_with_unique_keys s

    inner join {{ ref('snowplow_attribution_campaign_attributions') }} c

    on c.campaign = s.campaign and s.spend_tstamp < cv_tstamp

    and s.spend_tstamp > {{ snowplow_utils.timestamp_add('day', -90, 'cv_tstamp') }}
```

Next the model is aggregating data from both the channel and campaign attributions table by conversion, summing up all the different kinds of attribution (first touch, last touch etc.)

Here comes the second time based filter which you might want to adjust in case you run your package incrementally as default. If you adjusted the `snowplow__conversion_window_start_date` and `snowplow__conversion_window_end_date` and ran a specific conversion period, then you can leave it as per default, but for incremental package runs you might want to change the `snowplow__conversion_window_days` variable to extend the analysis period (by default it means it will process last 30 number of days since the last path being available in the paths source (most likely the last page view from the snowplow\_unified.views table)).

```jinja2
  {% if not var('snowplow__conversion_window_start_date') == '' and not var('snowplow__conversion_window_end_date') == '' %}

    cv_tstamp >= '{{ var("snowplow__conversion_window_start_date") }}' and cv_tstamp < '{{ var("snowplow__conversion_window_end_date") }}'

  {% else %}

    cv_tstamp >= {{ snowplow_utils.timestamp_add('day', -var("snowplow__conversion_window_days"), last_processed_cv_tstamp) }}

  {% endif%}
```

Finally, the model takes the aggregated conversion level data and does a series of unions to calculate metrics by attribution type (first touch, last touch etc.) Here again you might want to adjust what specific metrics you are interested in, add your own etc.

## Running the Attribution Analysis with a specific conversion period only

By default the Attribution package is designed to be run incrementally.

In case you only need ad hoc analysis studying only a specific conversion window and perhaps looking at a larger lookback period to gather a broader user journey, for cost saving purposes you might want to define a set period every time the model is run (e.g. on a monthly basis).

In that case, you can simply overwrite both the `snowplow__conversion_window_start_date` and `snowplow__conversion_window_end_date` variables to decide on the period. In this case the default `snowplow__conversion_window_days` variable will be disregarded by the package.

Bear in mind that it needs a full-refresh every time you process the package:

```bash
 dbt run --select snowplow_attribution --full-refresh --vars 'snowplow__allow_refresh: true'
```

## Bringing your own sources - customizing the paths\_to\_conversions() macro

There may be cases where you opt for not relying on the snowplow-unified based source tables. The main thing to care about is that you have a joined user\_id in both the conversion and path source.

There are two ways to go about this:

#### 1. Align your data sources in a way to fit the package logic

- create a view on top of your source data to make sure you align the expected input fields

- have the common user\_id field called `stitched_user_id` in both your path and conversions source view

- make sure to set the variable `snowplow__conversion_stitching` to True (it means the package will rely on the `stitched_user_id` fields for the joins)

#### 2. Align the package logic to work with your data sources

The `paths_to_conversions()` macro to your package is provided in the form of a dispatch macro that you can simply overwrite by adding a new macro with the same name in your dbt project.

The default structure of the macro is the following:

```sql
--

with paths as (

  ...

)



, conversions as (

  ...

)



, string_aggs as (

  ...

)



, arrays as (

  ...

)



{{ transform_paths('conversions', 'arrays') }}



select ...
```

You would most probably need to touch the `paths` and `conversions` cte to adjust how your data is getting processed incrementally from your custom sources. The `string_aggs` cte creates the individual paths the user travelled through to get to the conversion. These path strings are then converted to `arrays` in the subsequent cte.

After that there is a separate macro called `transform_paths` that is being called to provide the sql for the path transformations. It is quite complex, we would not be recommending to overwrite these.

After that the macro just selects the final paths\_to\_conversion data that will be used for the incremental update for a given run for this derived table.

---

# Snowplow Attribution dbt package
> Marketing attribution analysis with Snowplow dbt package supporting first-touch, last-touch, position-based, and linear attribution models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/

> **Info:** You will need Unified [version 0.4.0](https://github.com/snowplow/dbt-snowplow-unified/releases/tag/0.4.0) to use Attribution version 0.2.0.

​ **The package source code can be found in the [snowplow/dbt-snowplow-attribution repo](https://github.com/snowplow/dbt-snowplow-attribution), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-attribution/#/overview/snowplow_attribution).** ​

> **Tip:** Although there is no direct dependency, by default the attribution package is dependent on the `snowplow_unified_views` as well as the `snowplow_unified_conversions` table created by the [snowplow\_unified](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) dbt package. As long as the field names are matched you can use other source tables, too.

## Overview

​ The Snowplow Attribution dbt package produces a set of incremental derived tables to provide a basis for in-depth **Marketing Attribution Analysis** on an ongoing basis. It allows you to attribute the value of a conversion to one or more channels and campaigns depending on the conversion pathway. As a result, it becomes possible to determine the revenue per pathway (channel or campaign), as well as ROAS (Return On Ad Spend - the amount of revenue that is earned for every dollar spent on advertising) once you have cost data for each marketing channel or campaign.

This package consists of a series of dbt models that produce the following tables:

- `snowplow_attribution_paths_to_conversion`: Customer id and the paths the customer has followed that have lead to conversion
- `snowplow_attribution_campaign_attributions`: By campaign path and conversion level incremental table that attributes the conversion value based on various algorithms
- `snowplow_attribution_channel_attributions`: By channel path and conversion level incremental table that attributes the conversion value based on various algorithms
- `snowplow_attribution_overview`: The user defined report view (potentially showing ROAS)
- `snowplow_attribution_path_summary`: For each unique path, a summary of associated conversions, optionally non-conversions and revenue
- `snowplow_attribution_paths_to_non_conversion`: Customer id and the paths the customer has followed that have not lead to conversion. Optional drop and recompute table, disabled by default.

In the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) section you will find a step-by-step guide on how to operate the package as a whole.

![Package flow](/assets/images/attribution_package_flow_light-d9ba3bb672a244629720bba7291a9895.png)

### Attribution Models

The package currently offers 4 different attribution models all of which are calculated by default into both the incremental base tables and the report tables. If you would like to filter on them for reporting you can do so with `var('snowplow__attribution_list')`. Please note, however, that they will still be available for the incremental base tables.

- `first_touch`: Assigns 100% attribution to the first channel in each path.
- `last_touch`: Assigns 100% attribution to the last channel in each path.
- `position_based`: The first and last channels get 40% of the credit each, with the remaining channels getting the leftover 20% distributed evenly.
- `linear`: Assigns attribution evenly between all channels on the path. ​

![Data processing model for the attribution package](/assets/images/attribution_models_light-9fcb879888bac68c8da798cde1c97e9d.png)

## What is Marketing Attribution?

Marketing attribution determines which marketing tactics are contributing to sales or conversions by analysing the marketing touchpoints a consumer encounters on their journey to purchase. The aim is to determine which channels and marketing campaigns had the greatest impact on the decision to convert. There are many popular attribution models used by marketers which give insight into customers' behaviors, more specifically the pathways they took to purchase the product or service. This allows marketing teams to improve ROAS by changing marketing strategies and campaigns.

## Benefits of using our package

The package was implemented with a **glassbox philosophy** in mind, making it very **transparent** for data consumers how the data is transformed. It is processed **all in SQL** inside your warehouse, there are no external dependencies e.g. extra Python packages or black-box ML algorithms.

It is also very **flexible**, there are no limitations on the touchpoints (it works on all channels or campaigns). You can use ad spend data from any 3rd party data sources. Although there are no direct dependencies on data sources either, it is recommended to be used together with the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) dbt package for a seamless setup without the need to overwrite default dbt macros.

It also does the heavy-lifting for you and provides you with **incremental data models** reducing unnecessary costs even if you would like to regularly analyze the data based on a shifting time window in your reports (e.g. last 30 days).

## How to do Marketing Attribution with this package?

The purpose of this package is to allow an incremental, efficient way to do marketing attribution without having to read all your data every time.

In the below guide we will walk you through the data transformation process step-by-step in order for you to see how the source data changes downstream. This will give you and your team a transparent and easy-to-understand way to see how this package will lead you to valuable insights.

## Sources you are going to need

### 1. Conversion source

You will also need a table where the conversion events are stored. If you use the snowplow\_unified model and configure conversions to be modelled you will have this information in your `derived.snowplow_unified_conversions` table by default.

In case you store your conversion events elsewhere in the data warehouse (e.g. because those are transactional data that you decide you do not want to process through Snowplow), you can mutate that source table into the format of the conversions table (expand the Conversion formatting requirements section below for a list of all the conversion table columns) generated by the unified package, store it as a sql view, and refer to that in the variable `snowplow__conversions_source` (in the format of `schema.view_name`).

You could also union multiple sources (e.g. one transactional event data for automatic conversion events not dictated by user actions and then the rest, ending up in the unified\_conversions table). Bear in mind though, that the incremental logic is dictated based on the last processed conversion in your conversion source, you need to make sure those tables are synced before the model is run.

Please bear in mind, if you would like to calculate the attribution for conversion events not processed by Snowplow, you would need to make sure these events contain a user\_id type field which the package can rely on to find previous website visits for it to be able to attribute them to these conversions. Otherwise, your overall revenue will be more than the total attributed revenue.

**Conversion formatting requirements**

Fields to use (including the name) when creating the view:

- `cv_id`: a unique identifier for each conversion, can be a dummy UUID
- `event_id`: a unique identifier for the conversion event, can be dummy UUID
- `session_identifier` (OPTIONAL): can be left out if it is the only source or NULL for unions with the unified.conversions table
- `user_identifier`: the user id field to be used for identifying pre conversion website visits
- `user_id`: the user id field to be used for identifying pre-conversion website visits
- `stitched_user_id`: the user id field to be used for identifying pre conversion website visits (this will be used if conversion\_stiching is enabled)
- `cv_value`: the conversion amount
- `cv_tstamp`: a timestamp field when the conversion took place
- `dvce_created_tstamp` (OPTIONAL): a timestamp field, can be left out if it is the only source or the same as the cv\_tstamp for unions with the unified.conversions table
- `cv_type`: you can name the types of conversions to differentiate them, if needed, it should not be left NULL/blank
- `cv_tstamp_date`: (DATABRICKS/SPARK ONLY): the date of the `cv_tstamp` value

### 2. Path source

You will also need a source table to track your user journey / path with fields to be able to classify your marketing channels. The perfect table for this is your `derived.snowplow_unified_views` table (default):



Alternatively, you could use the `derived.snowplow_unified_sessions` table as well, but bear in mind that this will mean only the first channel/campaign will be counted within a session, and you will have to make sure the correct field reference is used in the `snowplow_attribution_paths_to_conversion` table by overwriting the `paths_to_conversion()` macro in your project (e.g `first_page_urlhost` instead of `page_urlhost`).

As for campaigns,

> **Tip:** To fully finish the config you might need to overwrite the `channel_classification()` macro. In case your classification logic for attribution analysis needs to be the same as the one already configured in the snowplow\_unified model you can simply leave the default macro which refers to that field. See the [channel\_group\_query macro](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/overridable-macros/) for more details.

### 3. Channel spend information (optional, but recommended)

You most likely have a warehouse with marketing (ad) spend information by channel and date, something like this:



To make it flexible to use what you already have, we suggest creating a view on top of the table you have, rename the fields that the model will use and add that view reference in `var('snowplow__spend_source')`. For more details on how to do this check out our [Quick Start Guide](/tutorials/attribution/intro/).

### 4. User mapping source

In order to be able to attribute the value of a conversion to one or more channels and campaigns it is essential to have a shared user identifier field between the conversion events and the preceding website visits. If the Unified dbt package is used for the conversion and conversion path source, there are two out-of-the box stitching options to use:

1. By enabling BOTH the `snowplow__view_stitching` the `snowplow__conversion_stitching` in the unified package, both sources will get the latest `stitched_user_id` field based on logged in user activity (see details [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/)). By setting the `snowplow__conversion_stitching` variable in the attribution package to true, the package will consider that field to take as a basis for both the conversion and the path source. Please note that this could potentially become a costly operation due to the volume of the snowplow\_unified\_views table, as after each run all the id fields will be updated by the latest mapping.

2. Alternatively (default from v.0.3.0 onwards), the package will rely on the `snowplow__user_mapping_source` defaulted to the `derived.snowplow_unified_user_mapping` table which will get used to take the latest logged in business user\_id field per user\_identifier, if available, otherwise it will keep the user\_identifier value for both the conversions and conversion path source before the two are joined.

3. When using custom sources or solutions, this logic could be overwritten in the dbt project using the `snowplow_attribution_paths_to_conversion()` dispatch macro. In case the optional `snowplow_attribution_paths_to_non_conversion` table is also in use, and it needs custom stitching logic, it is advised to disable it and create a custom model with the desired stitching logic.

Another option is if you alias your unique user id field as `stitched_user_id` in both your custom path and conversion source and set the `snowplow__conversion_stitching` variable to true, then the macro doesn't need to be altered but it will rely on these fields when linking conversions to paths.

### One-off setup

Once you have the sources ready you need to adjust the package settings that fit your business logic. This has to be done once, otherwise your data might be misaligned due to the incremental logic or you would need to run a full-refresh:

**Decide on your sessionization logic**

By default, Snowplow only considers the first pageview of a session important from an attribution point of view and disregards campaign information from subsequent page\_views. Google Analytics, on the other hand separates session as soon as campaign information is given. Set `var('consider_intrasession_channels')` variable to false in case you would like to follow Snowplow's logic, not GA's. If you opt for this calculation consider changing the `var('snowplow__conversion_path_source')` to `{{ target.schema }}.derived.snowplow_unified_sessions` for performance benefits.

Use the variable `snowplow__conversion_hosts` to restrict which hosts to take into account for conversions.

**Filter unwanted / wanted channels & campaigns**

You can specify a list of channels for the variable `snowplow__channels_to_exclude` to exclude them from analysis (if kept empty all channels are kept). For example, users may want to exclude the 'Direct' channel from the analysis.

You can also do the opposite, filter on certain channels to include in your analysis. You can do so by specifying them in the list captured within the variable `snowplow__channels_to_include`.

You can do either for campaigns, too, with the `snowplow__channels_to_exclude` and `snowplow__channels_to_include` variables.

#### **Transform paths**

In order to reduce unnecessarily long paths you can apply a number of path transformations that are created as part of user defined functions automatically in your warehouse by the package.

In order to apply these transformations, all you have to do is to define them in the `snowplow__path_transforms` variable as a dictionary. In case of `remove_if_last_and_not_all` and `remove_if_not_all` transformations, the transformation name is the key and a non-empty array is the value.

For other transformations (`exposure_path`, `first_path`, `unique_path`), no additional parameter is required, you can just use `null` as values. For more details on how to do this, check out the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/attribution/) E.g.: `{'exposure_path': null, 'remove_if_last_and_not_all': ['channel_to_remove_1', 'campaign_to_remove_1', 'campaign_to_remove_2']}` Please note that the transformations are applied on both campaign and channel paths equally.

**Path transform options**

Paths to conversion are often similar, but not identical. As such, path transforms reduce unnecessary complexity in similar paths before running the attribution algorithm. The following transformations are available: ​

1. **`exposure_path (default)`**: the same events in succession are reduced to one: `A → A → B` becomes `A → B`, a compromise between first and unique

2. **`unique_path`**: all events in a path are treated as unique (no reduction of complexity). Best for smaller datasets (small lookback window) without a lot of retargeting

3. **`first_path`**: keep only the first occurrence of any event: `A → B → A` becomes `A → B`, best for brand awareness marketing

4. **`remove_if_last_and_not_all`**: requires a channel to be added as a parameter, which gets removed from the latest paths unless it removes the whole path as it is trying to reach a non-matching channel parameter: E.g target element: `A` path: `A → B → A → A` becomes `A → B`

A common requirement is to take the **`last non-direct clicks`** only into account during attribution analysis. If you use the `remove_if_last_and_not_all` path transform with `direct` as the parameter to remove it from the end of the path (unless they are all direct) the `last-touch` attribution results should show the outcome you need. You can also filter out the direct channels altogether with the `snowplow__channels_to_exclude` variable.

5. **`remove_if_not_all`**: requires a channel to be added as a parameter, which gets removed from the path altogether unless it would result in the whole path's removal: E.g target element: `A` path: `A → B → A → A` becomes `B`

Apart from this, you can also restrict how far in time (`var('snowplow_path_lookback_days')`) and steps (`var('snowplow_path_lookback_steps')`) you want to allow your path to go from the actual conversion event.

> **Note:** Redshift users starting from 0.6.0 are only allowed to have one path transformation (e.g. either `exposure_path` or `first_path`). If using `remove_if_last_and_not_all` or `remove_if_not_all`, only single-item arrays are allowed. In case of other supported targets there are no such limitations, multiple path transformations can be applied, if neeeded.

#### **Other, macro based setup**

> **Tip:** To overwrite these macros correctly with those in your project, ensure you prefix the macro name by `default__` in the definition e.g.
>
> ```jinja2
> {% macro default__channel_classification() %}
>
>     case when (...)
>
> {% endmacro %}
> ```

#### `channel_classification` macro

> The [`channel_classification`](https://github.com/snowplow/dbt-snowplow-attribution/blob/main/macros/channel_classification.sql) macro is used to classify each marketing touchpoint from your path source deciding which marketing channel it was driven by. Be default it expects an already classified field called `default_channel_group`. In case the Attribution package is used together with the Unified Digital package, it is advisable to classify your pageviews and sessions upstream using the `channel_group_guery()` macro. For an in-depth guide on how to achive this check the [channel group query](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/overridable-macros/) macro documentation.

#### `attribution_overview` macro

> Defines the sql for the view called attribution overview which provides the main report calculating the return on advertising spend (ROAS). In order to do that you would need a marketing spend table as an additional source which will contain the spend information by channel and or campaign with a timestamp to filter on the period. If you are happy with the logic of the macro, you can just define your spend table in the `snowplow__spend_source` variable and let it run.

#### `paths_to_conversion` macro

> Macro to allow flexibility for users to modify the definition of the paths\_to\_conversion incremental table. By default the incremental table uses the `snowplow_unified_views` and `snowplow_unified_conversions` source tables but this macro allows for flexibility around that. You can also restrict which conversion events to take

#### **Adding tests (Optional)**

Please note that the Unified data model allows nulls on `user_identifier` field, but the Attribution package does not, by default we filter out nulls within the `paths_to_conversion` macro when joining on the `path_source` and also on the `conversion_source` using the `conversion_clause` variable, which by default includes the null filter. We encourage users to add their own tests in the appropriate level (e.g. in the derived.snowplow\_unified\_views level) to make sure you don't accidentally exclude those events without user\_identifier that are null (e.g. due to a tracking issue).

## Output

### Incremental data models to prepare for attribution analysis:

1. The **`derived.snowplow_attribution_paths_to_conversion`** model will aggregate the paths the customer has followed that have lead to conversion based on the path transformation and other limitations such as the path\_lookback\_step or path\_lookback\_days variable i.e. it combines the path and conversion source tables to produce an outcome. It looks like this:

| customer\_id | cv\_tstamp       | revenue | channel\_path                              | channel\_transformed\_path                 | campaign\_path                                    | campaign\_transformed\_path |
| ------------ | ---------------- | ------- | ------------------------------------------ | ------------------------------------------ | ------------------------------------------------- | --------------------------- |
| user\_id1    | 2022-06-11 15:33 | 20.42   | Direct                                     | Direct                                     | camp1 > camp2                                     | camp1 > camp2               |
| user\_id2    | 2022-07-30 11:55 | 24      | Direct > Direct                            | Direct                                     | camp1                                             | camp1                       |
| user\_id3    | 2022-06-08 20:18 | 50      | Direct > Direct                            | Direct                                     | camp2 > camp1                                     | camp2 > camp1               |
| user\_id1    | 2022-07-25 07:52 | 140     | Organic\_Search > Direct > Organic\_Search | Organic\_Search > Direct > Organic\_Search | Campaign 2 > Campaign 2 > Campaign 1 > Campaign 1 | Campaign 2 > Campaign 1     |

2. The **`derived.snowplow_attribution_channel_attributions`** unnests the paths from paths\_to\_conversion into their separate rows and calculates the attribution amount for that specific path step for each of the sql based attribution models:

| composite\_key        | event\_id | customer\_id | cv\_tstamp              | cv\_total\_revenue | channel\_transformed\_path       | channel         | source\_index | path\_length | first\_touch\_attribution | last\_touch\_attribution | linear\_attribution | position\_based\_attribution |
| --------------------- | --------- | ------------ | ----------------------- | ------------------ | -------------------------------- | --------------- | ------------- | ------------ | ------------------------- | ------------------------ | ------------------- | ---------------------------- |
| id1\_Video0           | event\_1  | user\_id1    | 2023-07-07 13:05:55.000 | 200                | Video                            | Video           | 0             | 1            | 200                       | 200                      | 200                 | 200                          |
| id2\_Display\_Other0  | event\_2  | user\_id2    | 2023-07-19 04:27:51.000 | 66.5               | Display\_Other > Organic\_Search | Display\_Other  | 0             | 2            | 66.5                      | 0                        | 33.25               | 33.25                        |
| id3\_Organic\_Search1 | event\_2  | user\_id2    | 2023-07-19 04:27:51.000 | 66.5               | Display\_Other > Organic\_Search | Organic\_Search | 1             | 2            | 0                         | 66.5                     | 33.25               | 33.25                        |

3. The **`derived.snowplow_attribution_campaign_attributions`** does the same, only for campaigns not channels.

### Drop and recompute reporting tables/views:

1. The **`derived.snowplow_attribution_path_summary`** shows the campaign/channel paths and the associated conversions (and optionally non-conversions, if the `path_to_non_conversions` table is enabled through its related variable `enable_path_to_non_conversions`)

| transformed\_path                     | conversions | non\_conversions | revenue |
| ------------------------------------- | ----------- | ---------------- | ------- |
| Direct                                | 3           | 25               | 94.42   |
| Organic\_Search                       | 2           | 26               | 206.5   |
| Referral > Direct                     | 0           | 1                | 0       |
| Video                                 | 1           | 2                | 200     |
| Organic\_Search > Paid\_Search\_Other | 0           | 2                | 0       |

2. The view called **`derived.snowplow_attribution_overview`** is tied to a dispatch macro of the same name which lets you overwrite it in your project, if needed. Given you specify your `var('snowplow__spend_source')` it will calculate the ROAS for you for each channel and campaign:

| path\_type | attribution\_type | touch\_point    | in\_n\_conversion\_paths | attributed\_conversions | min\_cv\_tstamp         | max\_cv\_tstamp         | spend   | sum\_cv\_total\_revenue | attributed\_revenue | roas     |
| ---------- | ----------------- | --------------- | ------------------------ | ----------------------- | ----------------------- | ----------------------- | ------- | ----------------------- | ------------------- | -------- |
| campaign   | first\_touch      | Campaign 2      | 2                        | 1                       | 2023-07-19 04:27:51.000 | 2023-07-25 07:52:34.000 | 100,000 | 206.5                   | 206.5               | 0.002065 |
| campaign   | last\_touch       | Campaign 1      | 3                        | 1                       | 2023-07-07 13:05:55.000 | 2023-07-30 11:55:24.000 | 100,000 | 364                     | 364                 | 0.00364  |
| channel    | first\_touch      | Display\_Other  | 1                        | 1                       | 2023-07-19 04:27:51.000 | 2023-07-19 04:27:51.000 | 100,000 | 66.5                    | 66.5                | 0.000665 |
| channel    | first\_touch      | Organic\_Search | 2                        | 0                       | 2023-07-19 04:27:51.000 | 2023-07-25 07:52:34.000 | 100,000 | 206.5                   | 0                   | 0        |

### Manifest table

We have included a manifest table to log information about the setup variables each time the **`paths_to_conversion`** incremental table runs to help prevent and debug issues.

---

# Keep cohesion between the Unified and Attribution packages
> Run Unified Digital and Attribution dbt packages together with proper lineage and source references for cohesive analysis.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/keeping-cohesion-with-unified/

In case there are data issues (e.g. you may want to reprocess the analysis based on the latest user stitching status) the Attribution package produced data models can be reprocessed fully without the need to refresh the Unified package, but not the other way round as it is used as one of the primary sources to rely on when creating the incremental tables. If the underlying data source changed which impacted the views or conversions source data, you would need to refresh the attribution package as well.

There is, however, no dependency between the packages when it comes to syncing them. You may want to run your Unified package once an hour for the more business critical tables to get updated, while you can process the Attribution package once a month, if that's what makes sense for your business. There is no concept of backfill limits or partials reprocessing for the Attribution package, regardless of the last run, the package will reprocess all the data since the last processed conversion event that became available in the source tables.

### Running both Unified and Attribution dbt packages from the same project

It might be practical to run both the Unified Digital and Attribution dbt packages from the same project.

When specifying the sources, make sure you change the default source references to `ref('...')` instead of hard hardcoding the `schema.table_name` for these variables. This allows dbt to establish a proper lineage graph to process things in order:

```yml
snowplow__conversion_path_source: "{{ ref('snowplow_unified_views') }}"

snowplow__conversions_source: "{{ ref('snowplow_unified_conversions') }}"

snowplow__user_mapping_source: "{{ ref('snowplow_unified_user_mapping') }}"
```

Keep in mind that each model keeps its own state. This means that you can, for example, set a different `start_date` for the attribution model if you add it at a later point.

```yml
vars:

  snowplow_unified:

    snowplow__start_date: '2024-06-03'

  snowplow_attribution:

    snowplow__attribution_start_date: '2024-09-03'
```

---

# Snowplow Ecommerce dbt package
> Transform raw ecommerce event data into derived tables for carts, checkouts, products, transactions, and sessions.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-ecommerce repo](https://github.com/snowplow/dbt-snowplow-ecommerce), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-ecommerce/#!/overview/snowplow_ecommerce).**

The package contains a fully incremental model that transforms raw ecommerce event data generated by the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) or [Mobile trackers](/docs/sources/mobile-trackers/) into a set of derived tables based around the following ecommerce data objects: carts, checkouts, products and transactions.

The Snowplow ecommerce data model aggregates Snowplow's out of the box `snowplow_ecommerce_action` events to create a set of derived tables based on the ecommerce data objects - carts, checkouts, products, transactions, users - that contain many useful dimensions as well as calculated measures.

## Overview

The package contains multiple staging models however the mart models are as follows:

- Base: Performs the incremental logic, outputting the table `snowplow_ecommerce_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model, and is the foundation for all other models generated.
- Carts: Parses the cart interactions that occur to provide handy filters and aggregations, which helps identify what happened to carts on a session level to extract, for example, abandoned carts with ease. The Carts module outputs the tables `snowplow_ecommerce_cart_interactions`.
- Checkouts: Parses checkout steps that occur to provide handy filters and aggregations to help identify which checkout steps were walked through, and what details were entered in each of these steps. This lends itself well to a funnel analysis. The Checkouts module outputs the `snowplow_ecommerce_checkout_interactions` table.
- Products: Parses product view and list information to provide insights into which products were being viewed, what details were being shown to the end user and how the user then interacted with these products. The Products module outputs the `snowplow_ecommerce_product_interactions` table.
- Transactions: Parses transaction actions to provide insights into which transactions occurred, how much revenue was generated from them, and other insights leveraging the many properties of the transaction ecommerce context. The Transactions module outputs the `snowplow_ecommerce_transaction_interactions` table.
- Sessions: Aggregates all other data into a sessions table which leverages the `domain_sessionid`, outputting the `snowplow_ecommerce_sessions` table.

## Mixing web and mobile events

The package makes no distinction between events tracked from the web and those tracked from a mobile application, so long as you are tracking `snowplow_ecommerce_action` events and from allowed `app_id`s.

The `sessionId` from the `client_session` context, and the `id` from the `mobile_screen` context, overwrite the `domain_sessionid` and `page_view_id` fields respectively in our intermediate and derived tables, for events where they are populated.

The configuration is different depending on what event sources you have:

- Web events only: the package will work out the box
- Mobile events only: the `snowplow__enable_mobile_events` package variable must be set to `true`, and the [`webPage` context](/docs/sources/web-trackers/tracker-setup/initialization-options/) must be available in your warehouse (even though it will not be populated for these mobile events)
- Both web and mobile events: the `snowplow__enable_mobile_events` package variable must be set to `true`

---

# Snowplow Identities dbt package
> Transform raw identity entities and merge events into derived tables for identity resolution, identifier mapping, and audit.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-identities-data-model/

**The package source code can be found in the [snowplow-incubator/dbt-snowplow-identities](https://github.com/snowplow-incubator/dbt-snowplow-identities/) repository.**

The Snowplow Identities dbt package transforms the raw [identity entities and merge events](/docs/identities/concepts/#data-types) produced by [Identities](/docs/identities/) into a set of derived tables for identity resolution, identifier lookup, and audit.

The package supports Snowflake and BigQuery.

Check out the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/identities/) and [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/identities/) pages to get started.

> **Note:** The package uses a simplified, timestamp-based incremental strategy rather than the session-based incremental logic used by other Snowplow dbt packages.
>
> This means that the standard guidance around [package mechanics](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/), [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/), and [dbt operations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) doesn't apply to this package.

## Key fields

The Identities models use these field names consistently:

| Field                  | Description                                                                                                                                                                                                                               |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `snowplow_id`          | The Snowplow ID enriched onto the event by the Identities service in real time. Represents the identity at the time the event occurred. It may later be merged into a different ID.                                                       |
| `active_snowplow_id`   | The current, unified identifier that represents a cluster of identities. When identities merge over time, this field always points to the surviving parent ID in the cluster. One `active_snowplow_id` maps to many `snowplow_id` values. |
| `previous_snowplow_id` | Used in the ID changes table to denote the `snowplow_id` that was merged into another ID.                                                                                                                                                 |

## Output models

The package produces six output models.

### `snowplow_identities_id_changes`

A fact table containing a complete history of all changes to Snowplow ID clusters, including both merges and new ID creation. Use this table for auditing the full lifecycle of identity changes.

| Column                 | Description                                                                             |
| ---------------------- | --------------------------------------------------------------------------------------- |
| `id_change_key`        | Surrogate primary key (hash of `snowplow_id` + `previous_snowplow_id` + `effective_at`) |
| `snowplow_id`          | The Snowplow ID after this change                                                       |
| `previous_snowplow_id` | The ID that was merged; `NULL` for new ID creation                                      |
| `effective_at`         | Event timestamp; `derived_tstamp` for creates, `merged_at` for merges                   |
| `changed_at`           | Processing timestamp, recording when dbt materialized this record                       |
| `change_type`          | Either `created` or `merged`                                                            |
| `first_seen_event_id`  | The first event that triggered this change                                              |
| `first_seen_app_id`    | The `app_id` where this change was first observed                                       |

### `snowplow_identities_snowplow_id_mapping`

A mapping between any previous `snowplow_id` and its current `active_snowplow_id`. The model automatically resolves chains of merges, so you always get the most current unified ID. Only merged `snowplow_id` values appear in this table — unmerged IDs are their own `active_snowplow_id` by definition.

| Column               | Description                                                            |
| -------------------- | ---------------------------------------------------------------------- |
| `active_snowplow_id` | The current, canonical Snowplow ID                                     |
| `snowplow_id`        | A merged Snowplow ID currently represented by the `active_snowplow_id` |
| `merged_at`          | Timestamp of when this mapping was last modified                       |
| `model_tstamp`       | Timestamp of the dbt run that produced this record                     |

To look up the `active_snowplow_id` for records in another table, use a left join with a `COALESCE` fallback:

```sql
select

    s.*,

    coalesce(m.active_snowplow_id, s.snowplow_id) as active_snowplow_id

from some_other_model s

left join snowplow_identities_snowplow_id_mapping m

    on s.snowplow_id = m.snowplow_id
```

The `COALESCE` handles the case where a `snowplow_id` has not been merged — it is still its own `active_snowplow_id`.

### `snowplow_identities_id_mapping_scd`

A slowly changing dimension (SCD) type 2 table that captures how the canonical Snowplow ID mapping changes over time. This is useful for audit purposes, and reporting that requires the state of the identity graph at a specific point in time.

| Column               | Description                                                                              |
| -------------------- | ---------------------------------------------------------------------------------------- |
| `active_snowplow_id` | The canonical Snowplow ID for this time period.                                          |
| `snowplow_id`        | A Snowplow ID represented by the `active_snowplow_id` (including the active one itself). |
| `effective_at`       | Timestamp from which this record is valid.                                               |
| `superseded_at`      | Timestamp at which this record was superseded. `NULL` for the current record.            |
| `is_current`         | Boolean flag; `true` when `superseded_at` is `NULL`.                                     |

To query the state of the identity graph at a specific point in time:

```sql
select *

from snowplow_identities_id_mapping_scd

where effective_at <= '2026-01-15'

  and (superseded_at > '2026-01-15' or superseded_at is null)
```

### `snowplow_identities_identifier_mapping`

Contains all external identifiers linked to an `active_snowplow_id`. The list of identifier types is [configurable](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/identities/#4-configure-your-identifiers) and should reflect the identifiers you have configured in Console, plus any additional identifiers you want available in the output tables. Identifiers are stored in a normalized format with `id_type` and `id_value` columns rather than pivoted into separate columns per type.

Use this table to look up addressable identifiers, such as email, that can be activated in downstream systems.

| Column                | Description                                                                                               |
| --------------------- | --------------------------------------------------------------------------------------------------------- |
| `active_snowplow_id`  | The current, canonical Snowplow ID                                                                        |
| `id_type`             | The type of identifier e.g. `email`, `user_id`, `domain_userid`                                           |
| `id_value`            | The actual identifier value, or its SHA-256 hash if [hashing is enabled](/docs/identities/configuration/) |
| `first_seen_event_id` | The first event that contributed this identifier                                                          |
| `first_app_id`        | The `app_id` of the event where this identifier was first observed                                        |
| `last_app_id`         | The `app_id` of the event where this identifier was most recently observed                                |
| `first_seen_at`       | Timestamp of the event where this identifier was first observed                                           |
| `last_seen_at`        | Timestamp of the event where this identifier was last observed                                            |
| `uuid`                | Surrogate key                                                                                             |

### `snowplow_identities_new_identities`

One row per `snowplow_id`, recording the first and last observed activity for that identity. The [`snowplow__identifiers`](/docs/identities/configuration/) variable determines which identifier columns appear in this table.

| Column                 | Description                                                              |
| ---------------------- | ------------------------------------------------------------------------ |
| `snowplow_id`          | The Snowplow ID; unique key for this table                               |
| `created_at`           | Timestamp when this identity was first created, from the identity entity |
| `first_seen_event_id`  | The first event associated with this identity                            |
| `first_app_id`         | The `app_id` where this identity was first observed                      |
| `last_app_id`          | The `app_id` where this identity was most recently observed              |
| `first_derived_tstamp` | `derived_tstamp` of the first event for this identity                    |
| `last_derived_tstamp`  | `derived_tstamp` of the most recent event for this identity              |
| _(identifier columns)_ | One column per configured identifier (e.g., `domain_userid`, `user_id`)  |

### `snowplow_identities_merge_events`

One row per identity merge event. Each row records the surviving (`active`) Snowplow ID and the full hierarchy of IDs that were merged into it. Use this table to audit the raw merge operations as they occurred.

| Column               | Description                                                                                                             |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `merge_event_id`     | The `event_id` of the `identity_merge` event — unique key for this table                                                |
| `active_snowplow_id` | The surviving Snowplow ID after this merge                                                                              |
| `collector_tstamp`   | Timestamp when the merge event was collected                                                                            |
| `derived_tstamp`     | Timestamp when the merge event occurred                                                                                 |
| `merged`             | Array containing the full merge hierarchy (all IDs merged into `active_snowplow_id`)                                    |
| `merges`             | Array of objects describing each individual merge operation, with `snowplow_id`, `merged_at`, and `triggering_event_id` |

---

# Snowplow Media Player dbt package
> Transform raw media player event data into derived tables for media plays, ad views, and media stats across HTML5, YouTube, and Vimeo.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-media-player repo](https://github.com/snowplow/dbt-snowplow-media-player), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-media-player/#!/overview/snowplow_media_player).**

The package contains a fully incremental model that transforms raw media player event data into derived tables for easier querying. It can support media events tracked using the following tracking implementations on Web and mobile:

- on Web using plugins for our [JavaScript trackers](/docs/sources/web-trackers/):

  - [media plugin](/docs/sources/web-trackers/tracking-events/media/) that can be used to track events from any media player.
  - [HTML5 media tracking plugin](/docs/sources/web-trackers/tracking-events/media/html5/).
  - [YouTube tracking plugin](/docs/sources/web-trackers/tracking-events/media/youtube/).
  - [Vimeo tracking plugin](/docs/sources/web-trackers/tracking-events/media/vimeo/).

- [media tracking APIs on our iOS and Android trackers](/docs/sources/mobile-trackers/tracking-events/media-tracking/) for mobile apps.

- [media tracking APIs on our Roku tracker](/docs/sources/roku-tracker/media-tracking/) for [Roku](https://www.roku.com/) devices

**Version 1 and version 2 of the media event and context entity schemas**

There are two versions of schemas for media events that our trackers may use to track media events. This has an effect on the information provided by the media package. In contrast with v1, the v2 schemas contain information about the media playback that is computed directly on the tracker and is more accurate (e.g., play time, buffering time). They also introduce new schemas for tracking ads during media playback.

1. v1 media schemas (used by the HTML5 and YouTube plugin for JavaScript tracker and the Roku tracker):

   - [media-player event schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player_event/jsonschema/1-0-0) used for all media events.

   - [media-player context v1 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0).

   - Depending on the plugin / intention there are player-specific contexts:

     - in case of embedded YouTube tracking: Have the [YouTube specific context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.youtube/youtube/jsonschema/1-0-0) enabled.
     - in case of HTML5 audio or video tracking: Have the [HTML5 media element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/media_element/jsonschema/1-0-0) enabled.
     - in case of HTML5 video tracking: Have the [HTML5 video element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/video_element/jsonschema/1-0-0) enabled.

2. v2 media schemas (used by the media and Vimeo plugins for the JavaScript trackers, the Roku tracker, and the mobile trackers):

   - [per-event media event schemas](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.media).
   - [media-player context v2 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0).
   - optional [media-session context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0).
   - optional [media-ad](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0) and [ad break](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0) context schema.

> **Note:** Support for the version 2 schemas (as used by the media, Vimeo JS plugin, Roku tracker or the mobile trackers) has been added in version 0.6 of the media player package. Older package versions only support version 1 schemas.

## Overview

The package contains multiple staging models however the mart models are as follows:

- **Base:** Performs the incremental logic, outputting the table `snowplow_media_player_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model, and is the foundation for all other models generated.

- **Media base:** Summarizes the key media player events and metrics of each media element on a media session level (or media\_id and page/screen view if media session context is not tracked, which is essentially the same), identified by the play\_id key, which is considered as a base aggregation level for media events.

  - The media base module outputs the `snowplow_media_player_base` table.
  - It also produces a `_pivot_base` table to calculate the percent\_progress boundaries and weights that are used to calculate the total play\_time and other related media fields.

- **Media plays:** Removes impressions from the media base table while keeping the same structure as the `snowplow_media_player_base` table. Removing impressions means that only media sessions (identified by the `play_id` key) in which the user played the content are kept.
  - The media plays module outputs the `snowplow_media_player_plays_by_pageview` view.

- **Media stats:** Aggregates the media base table to individual `media_identifier` level, calculating the main KPIs and overall video/audio metrics.

  - It uses the native dbt incremental materialization on a pageview basis after a set time window passed. This is to prevent complex and expensive queries due to metrics which need to take the whole page\_view events into calculation. This way the metrics will only be calculated once per pageview / media, after no new events are expected.
  - The media stats module outputs the `snowplow_media_player_media_stats` table.

- **Media ad views:** Summarizes metadata and KPIs for each ad view within a media playback. Each ad a user viewed will result in a single row with information about the progress reached, whether the ad was skipped or clicked.
  - The media ad views module outputs the `snowplow_media_player_media_ad_views` table.

- **Media ads:** Aggregates all ad views to produce a summary of the KPIs for each ad played within each media content. Produces total counts of views, skips, clicks and progress reached. It also counts the metrics in terms of unique users (e.g., number of users who clicked the ad).
  - It outputs the `snowplow_media_player_media_ads` table.

## Identifiers

The package generates robust identifiers for use in the incremental logic and keys for the derived tables.

| Identifier           | Definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `session_identifier` | The session identifier as defined in your dbt project variables which is is used for the package's [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/). If not set, this defaults to the `mediaSessionId` from the media session entity if enabled, else to the page/screen view id. This ensures all media events for a play spanning multiple `domain_sessionid`s are modelled. The more traditional session identifier `domain_sessionid` can be extracted from the `domain_sessionid_array` field. |
| `user_identifier`    | The user identifier as defined in your dbt project variables. If not set, this defaults to be `domain_userid` for web or the `userId` from the client session entity for mobile.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `media_identifier`   | Used to identify individual media elements/content. It is generated from the `player_id`, `media_label`, `media_type` and `media_player_type` fields.                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `play_id`            | The unique identifier for each individual play of media content. This is the `mediaSessionId` from the media session entity if enabled. Otherwise this uses the page/screen view identifier together with the `media_identifer` to generate a unique play id.                                                                                                                                                                                                                                                                                                                         |
| `media_ad_id`        | Generated identifier that identifies an ad (identified using the `ad_id`) played with a specific media (identified using the `media_identifier`) and on a specific platform (based on the `platform` property).                                                                                                                                                                                                                                                                                                                                                                       |
| `media_ad_view_id`   | The unique identifier for each individual ad view. It is generated from the `play_id`, `ad_break_id` and `media_ad_id` fields.                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

## Mixing web and mobile events

The package makes no distinction between events tracked from the web and those tracked from a mobile application, so long as you are tracking media events and from allowed `app_id`s. The `sessionId` from the `client_session` context, and the `id` from the `mobile_screen` context, overwrite the `domain_sessionid_array` and `page_view_id` fields respectively in our intermediate and derived tables, for events where they are populated. If you are just using web events, the package will work out the box. If you are using a mix of web and mobile events, you will need to set the `snowplow__enable_mobile_events` package variable to `true` and events will be processed from both sources. If you are only tracking mobile events, you can set the `snowplow__enable_web_events` to `false`.

## Custom models

There are two custom models included in the package which could potentially be used in downstream models:

1. the `snowplow_media_player_session_stats` table, which aggregates the `snowplow_media_player_base` table on a session level

2. the `snowplow_media_player_user_stats` table, which aggregates the `snowplow_media_player_session_stats` to user level

By default these are disabled, but you can enable them in the project's `profiles.yml`, if needed.

```yml
  models:

    snowplow_media_player:

      custom:

        enabled: true
```

Just like in case of all our packages, users are encouraged to use the Media Player model and its incremental logic to design their own custom models / modules. The `snowplow_media_player_base_events_this_run` table is designed with this in mind, where a couple of potentially useful fields are generated that the Media Player model does not use downstream but they nonetheless have the potential to be incorporated into users custom models. For more information on creating custom models, [visit the guide here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/).

---

# Snowplow Normalize dbt package
> Flatten and normalize self-describing events and entities into dedicated tables for downstream ETL and analysis tools.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/

> **Note:** Normalize in this context means [database normalization](https://en.wikipedia.org/wiki/Database_normalization), as these models produce flatter data, not statistical normalization.

**The package source code can be found in the [snowplow/dbt-snowplow-normalize repo](https://github.com/snowplow/dbt-snowplow-normalize), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-normalize/#/overview/snowplow_normalize).**

The package provides [macros](https://docs.getdbt.com/docs/build/jinja-macros) and a python script that is used to generate your normalized events, filtered events, and users table for use within downstream ETL tools. See the [Model Design](#model-design) section for further details on these tables.

The package only includes the base incremental scratch model and does not have any derived models, instead it generates models in your project as if they were custom models you had built on top of the [Snowplow incremental tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), using the `_this_run` table as the base for new events to process each run. See the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) section for the variables that apply to the incremental model.

> **Note:** The incremental model is simplified compared to the standard unified model, this package does not use sessions to identify which historic events to reprocess and just uses the `snowplow__session_timestamp` (defaults to `collector_tstamp`) and package variables to identify which events to (re)process.

## Model Design

![Data processing model for the normalize package](/assets/images/normalized_model_light.drawio-3782b7eca528cbfd3ea13194b8189e03.png)

The package outputs models for 3 types of models: normalized events, a filtered events model, and a users model (different to the users table generated by the `web` or `mobile` packages).

### Normalized Event Models

For each `event_names` listed, a model is generated for records with matching event type(s) and with the specified columns from the `atomic.events` table, including flattened (1 level) versions of your [self-describing event](/docs/fundamentals/events/#self-describing-events) or [context](/docs/fundamentals/entities/) columns. The table names are versioned based on the same versioning that is used to handle schema evolution with your data warehouse tables (see [Updating Your Models](#updating-your-models) section for potential issues with evolving schemas). In the case of multiple event types for a table you will need to provide the name and version of the table yourself. These can be combined into other models you create, or be used directly in downstream tools. The model file itself consists of lists of variables and a macro call.

For example, if you have 3 `event_names` listed as `['page_view']`, `['page_ping']`, and `['link_click', 'deep_link_click']` then 3 models will be generated, each containing only those respective events from the atomic events table.

### Filtered Events Model

A single model is built that provides `event_id`, `snowplow__partition_tstamp` (defaults to `collector_tstamp`), and the name of the Normalized Event Model that the event was processed into, it does not include records for events that were not of an event type in your configuration. The model file itself is a series of `UNION` statements.

### Users Model

The users model provides a more traditional view of a Users table than that presented in the other Snowplow dbt packages. This model has one row per non-null `user_id` (or other identifier column you specify), and takes the latest (based on `collector_tstamp`) values from the specified contexts to ensure you always have the latest version of the information that you choose to collect about your users. This is designed to be immediately usable in downstream tools. The model file itself consists of lists of variables and a macro call.

## Package Overview

This package consists of two macros, a python script, and some example configuration files to help you get started, as well as the incremental models to build these normalized tables from:

- `normalize_events` _(macro)_: This macro does the heavy lifting of the package, taking a series of inputs to generate the SQL required to normalize the `atomic.events` table and flatten any [self-describing event](/docs/fundamentals/events/#self-describing-events) or [context](/docs/fundamentals/entities/) columns. While you can use this macro manually it is recommended to create the models that use it by using the script provided.

- `users_table` _(macro)_: This macro takes a series of inputs to generate the SQL that will produce your users table, using the user identifier column you define, and any custom contexts from your events table. The identifier will be always be renamed to `user_id` in the output.

- `snowplow_normalize_model_gen.py` _(script)_: This script uses an input configuration to generate the models based on a provided configuration file. See the [operation](#operation) section for more information.

  ```yml
  usage: snowplow_normalize_model_gen.py [-h] [--version] [-v] [--dryRun] [--configHelp] [--cleanUp] config



  Produce dbt model files for normalizing your Snowplow events table into 1 table per event type.



  positional arguments:

  config         relative path to your configuration file



  optional arguments:

  -h, --help     show this help message and exit

  --version      show program's version number and exit

  -v, --verbose  verbose flag for the running of the tool

  --dryRun       flag for a dry run (does not write/delete any files)

  --configHelp   prints information relating to the structure of the config file

  --cleanUp      delete any models not present in your config and exit (no models will be generated)
  ```

- `example_normalize_config.json`: This file is an example of an input to the python script, showing all options and valid values. For additional information about the file structure run `python dbt_packages/snowplow_normalize/utils/snowplow_normalize_model_gen.py --configHelp` in your project root.

- `example_resolver_config.json`: This file is an example [Iglu Resolver](/docs/api-reference/iglu/iglu-resolver/) configuration. It supports custom iglu servers with API keys, but does not currently support accessing embedded registries. For more information please see the Resolver docs.

- `models/base/`: Models relating to the incremental nature of the package, processing only new events (and those covered by the lookback window).

## Operation

> **Tip:** If you are using dbt Cloud then you will need to run the model-generating script locally, as there is no way to do this in cloud, and then copy your models up to your Cloud environment. You'll still need to add the package to your `packages.yml` file in your project.
>
> You will need to have [python installed](https://www.python.org/downloads/), and basic experience using the terminal and a text editor, but there is no need to install dbt. If you clone or download the [package repo](https://github.com/snowplow/dbt-snowplow-normalize) you can follow the instruction below, but remove the `dbt_packages/snowplow_normalize` from the front of paths when you see it.

In general, it should only be required to run the script in this package once to begin with, then only as/when you need to add new models based on new events or alter the contexts attached to existing ones. While it is possible to manually set the values and use the macros, it is not recommended due to the time it would take and the likelihood of making mistakes.

The script should always be run from the **root** of your dbt project (the same level your `dbt_project.yml` file is at).

### Install python packages

> **Note:** Python versions between 3.7 and 3.10 (inclusive) are currently supported

The script only requires 2 additional packages (`jsonschema` and `requests`) that are not built into python by default, you can install these by running the below command, or by installing them by your preferred method.

```bash
pip install -r dbt_packages/snowplow_normalize/utils/requirements.txt
```

### Configuration File

> **Note:** There may be changes to the config file between versions, this page will always contain the latest information but if you are using an older version you should call the python script with the `--configHelp` flag.

The configuration file is used to provide the information needed to generate the SQL models listed above. We use the [schemas](/docs/fundamentals/schemas/) that are defined in your event tracking to ensure full alignment with the data and reduce the amount of information you need to provide.

> **Info:** The config file is a JSON file which can be viewed by running the python script with the `--configHelp` flag. The config file can be located anywhere in your project, but it must have the following structure. Note that you must provide **at least** one of `event_columns`, `self_describing_event_schemas` or `context_schemas` for each event listed.

**Field Description:**

- `config` _(required - object)_:

  - `resolver_file_path` _(required - string)_: Relative path to your resolver config json, or `"default"` to use [iglucentral](/docs/api-reference/iglu/iglu-repositories/#iglu-central) only
  - `filtered_events_table_name` _(optional - string)_: Name of filtered events table, if not provided it will not be generated
  - `users_table_name` _(optional - string)_: Name of users table, default `events_users` if user schema(s) provided
  - `validate_schemas` _(optional - boolean)_: If you want to validate schemas loaded from each iglu registry or not, default `true`
  - `overwrite` _(optional - boolean)_: Overwrite existing model files or not, default `true`
  - `models_folder` _(optional - string)_: Folder under `models/` to place the models, default `snowplow_normalized_events`
  - `models_prefix` _(optional - string)_: The prefix to apply (with `_`) to all normalize model table names, if a `table_name` isn't provided, default `snowplow`

- `events` _(required - array)_:

  - `event_1` _(required - object)_:

    - `event_names` _(required - array)_: Name(s) of the event type(s), value of the `event_name` column in your warehouse
    - `event_columns` _(optional (>=1 of) - array)_: Array of strings of flat column names from the events table to include to include in the model
    - `self_describing_event_schemas` _(optional (>=1 of) - array)_: `iglu:com.` type url(s) for the self-describing event(s) to include in the model
    - `self_describing_event_aliases` _(optional - array)_: array of strings of prefixes to the column alias for self describing events
    - `context_schemas` _(optional (>=1 of) - array)_: Array of strings of `iglu:com.` type url(s) for the context/entities to include in the model
    - `context_aliases` _(optional - array)_: Array of strings of prefixes to the column alias for context/entities
    - `table_name` _(optional if only 1 event name, otherwise required- string)_: Name of the model, default is the `event_name`
    - `version` _(optional if only 1 event name, otherwise required - string - length 1)_: Version number to append to table name. If (only one)`self_describing_event_schema` is provided the major version number from that is used, default `1`

  - `event_2` _(optional - object)_

  - ...

  - `event_n` _(optional - object)_

- `users` _(optional - if not provided will not generate users model - object)_

  - `user_id` _(optional - if not provided will use default `user_id` field - object)_

    - `id_column` _(required - string)_: Name of column or attribute in the schema that defines your user\_id, will be converted to a string in Snowflake
    - `id_self_describing_event_schema`_(optional - string)_: `iglu:com.` type url for the self-describing event schema that your user\_id column is in, used over `id_context_schema` if both provided
    - `id_context_schema`_(optional - string)_:`iglu:com.` type url for the context schema that your user\_id column is in
    - `alias`: _(optional - string)_: Alias to apply to the id column

  - `user_contexts` _(optional (>=1 of) - array)_: Array of strings of `iglu:com.` type url(s) for the context/entities to add to your users table as columns, if not provided will not generate users model

  - `user_columns` _(optional (>=1 of) - array)_: Array of strings of flat column names from the events table to include in the model

**Example Config File:**

```json
{

    "config":{

        "resolver_file_path": "utils/example_resolver_config.json",

        "filtered_events_table_name":"snowplow_events_normalized",

        "users_table_name":"snowplow_context_users",

        "validate_schemas": true,

        "overwrite": true,

        "models_folder": "snowplow_normalized_events",

        "models_prefix": "snowplow"

    },

    "events":[

        {

            "event_names": ["page_view", "page_ping"],

            "event_columns": [

                "domain_userid",

                "app_id",

                "load_tstamp",

                "page_url"

            ],

            "context_schemas":[

                "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",

                "iglu:nl.basjes/yauaa_context/jsonschema/1-0-4"

            ],

            "context_aliases": ["page", "yauaa"],

            "table_name": "snowplow_page_events",

            "version": "1"

        },

        {

            "event_names": ["link_click"],

            "event_columns": [

                "domain_userid",

                "app_id",

                "refr_urlpath"

            ],

            "self_describing_event_schemas": ["iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1"],

            "self_describing_event_aliases": ["click"],

            "context_schemas":[

                "iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0"

            ]

        }

    ],

    "users":{

        "user_id": {

            "id_column": "userId",

            "id_self_describing_event_schema": "iglu:com.google.analytics.measurement-protocol/user/jsonschema/1-0-0",

            "id_context_schema": "iglu:com.zendesk.snowplow/user/jsonschema/1-0-0",

            "alias": "true_user_id"

        },

        "user_contexts": ["iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0",

            "iglu:com.iab.snowplow/spiders_and_robots/jsonschema/1-0-0"],

        "user_columns": [

            "domain_userid",

            "network_userid",

            "app_id"

        ]

    }

}
```

***

An example configuration can be found in the `utils/example_normalize_config.json` file within the package.

> **Tip:** You should keep your configuration file, and your resolver file if you have one, at your project level and not inside the the snowplow-normalize package to avoid them being overwritten/deleted when the package is updated.

### Producing your models

To produce your models you need to run

`python dbt_packages/snowplow_normalize/utils/snowplow_normalize_model_gen.py path/to/your/config.json`

(or equivalent path in Windows) from the root of your dbt project. This will produce one `.sql` file for each of the event name specified in the `events` part of your configuration, one `.sql` file for the combined filtered events table if a name was provided, and one `.sql` file for your users table if schema(s) were provided. These files will be in your `models` folder in the sub-folder specified in your config.

> **Info:** Custom error messages have been added to the script to try and catch any issues and provide suggested resolutions to any issues such as invalid configurations or failing validation of schemas. If you persist in getting errors when validating schemas you believe you be correct, you can disable this validation by setting `validate_schemas` to `false` in your config.

### Adding new models

To add new models for new types of events, simply add them to your config and run the model again. Note that `overwrite` must be set to `true` for your filtered events table model to include these new events, and that this will overwrite any manual changes you made to the other models generated by this script.

### Updating your models

> **Warning:** Adding or removing columns from your dbt model, either manually or by changes caused by a different schema version, without specifying the `on_schema_change` model configuration or directly altering the tables in your database, is very likely to result in unintended outcomes - either your new columns will not appear or dbt may return an error. Please see the [dbt docs](https://docs.getdbt.com/docs/build/incremental-models#what-if-the-columns-of-my-incremental-model-change) for what to do in this situation. If you need to backfill new columns regardless you can perform a rerun of the entire model following the instructions [here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#refresh-only-some-models-in-a-package) _(note using the `--full-refresh` flag will not work in this case due to the use of the Snowplow incremental logic)_.

If you wish to update your models, such as adding a new context, removing a column, or updating to a new schema version, you must update your config file and run the script again, ensuring the `overwrite` value is set to `true`. This will remove any custom changes you made to the model file itself and you will need to re-add these.

All events models will be updated including the filtered events table, it is not possible at this time to update just a subset of models.

> **Info:** As your schemas for custom contexts and unstructured events evolve, multiple versions of the same column may be created in your events table (depending on the type of data loader being used) e.g. custom\_context\_1\_0\_0, custom\_context\_1\_0\_1. When modeling Snowplow data it can be useful to combine or coalesce each nested field across all versions of the column for a continuous view over time.
>
> The snowplow-normalize package handles this for you, all you have to do is make sure you add all the versions in the relevant config file parameter (`self_describing_event_schemas` / `context_schemas`):
>
> ```json
> "self_describing_event_schemas": ["iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-0-0", "iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-1-0"]
> ```
>
> The columns will then appear as part of the derived model configuration and will then be coalesced when compiled.
>
> ```jinja2
> {%- set sde_cols = ['UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_GEOLOCATION_CONTEXT_1_0_0', 'UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_GEOLOCATION_CONTEXT_1_1_0'] -%}
> ```

#### Removing specific columns

If you want to not select all columns from your self describing event or context then you can manually delete them from the lists in each model file, ensuring to delete the matching record in the `*_types` list as well. The same warning applies from [Updating your models](#updating-your-models), and any overwriting run of the script will remove these changes.

### Removing models

You can remove all models that don't exist in your config by running the script with the `--cleanUp` flag. This will scan the `models_folder` folder provided in your config and list all models in this folder that don't match those listed in your config file; you will then be asked to confirm deletion of these. If you may wish to re-enable these models at a later date you can [disable them in your project](https://docs.getdbt.com/reference/resource-configs/enabled) instead.

> **Warning:** If you have other models in the same sub-folder that were not generated via this package then this will attempt to delete those files.

---

# App errors module for Unified Digital
> Track and analyze application errors with the App errors module for the Unified Digital dbt package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/app-errors-module/

Assuming your tracker is capturing `application_error` events, the module can be enabled by configuring the `dbt_project.yml` file:

```yaml


vars:

snowplow_unified:

  snowplow__enable_app_errors: true

  
```

---

# Consent module for Unified Digital
> Model user consent preferences and GDPR compliance with the Consent module for the Unified Digital dbt package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/

This custom module is built as an extension of the dbt-snowplow-unified package.

It transforms raw `consent_preferences` and `cmp_visible` event data into derived tables for easier querying. These events are generated by the **Enhanced Consent plugin** of the [JavaScript tracker](/docs/sources/web-trackers/).

> **Info:** For the incremental logic to work within the module you **must** use at least `RDB Loader v4.0.0`, as the custom module relies on the additional `load_tstamp` field for dbt native incrementalisation.
>
> Whenever a new consent version is added to be tracked, the model expects an `allow_all` event in order to attribute the events to the full list of latest consent scopes. It is advisable to send a test event of that kind straight after deployment so that the model can process the data accurately.

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw consent tracking events:

- `snowplow_unified_consent_log`: Snowplow incremental table showing the audit trail of consent and Consent Management Platform (cmp) events.
- `snowplow_unified_consent_users`: Incremental table of user consent tracking stats.
- `snowplow_unified_consent_totals`: Summary of the latest consent status, per consent version.
- `snowplow_unified_cconsent_scope_status`: Aggregate of current number of users consented to each consent scope.
- `snowplow_unified_cmp_stats`: Used for modeling cmp\_visible events and related metrics.
- `snowplow_unified_consent_versions`: Incremental table used to keep track of each consent version and its validity.

## Operation

It is assumed that the Snowplow unified package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_consent` variable most conveniently set in your dbt\_project.yml file:

```yaml


vars:

snowplow_unified:

  snowplow__enable_consent: true

  
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_unified` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_unified` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Model conversion events with the Unified Digital package
> Model and track conversion events with the Unified Digital dbt package for attribution and funnel analysis.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/

Conversion events are a type of event that's important to your business, be that a transaction, a sign up to a newsletter, or the view of a specific page. Whatever type of event matters to you, so long as it can be determined from a single event record, you'll be able to model and aggregate these conversions at a session level with our package.

Because we know that each user may have a different concept of what a conversion means to them, for example it could be a specific type of event such as a `page_view` on a specific url, a self-describing event such as `sign_up`, or even a bespoke `conversion` event type with some attached context, the way we model conversions is also flexible and allows you to change your definition over time. This allows you to track multiple types of conversions, with varying logic, even retrospectively.

There are two ways that you can model conversions in the Unified package:

1. In the **sessions table only**
2. In the **optional conversions module**

## 1. Enabling conversions in the sessions table only

By configuring the conversion specifications through the `snowplow__conversion_events` variable you can enable conversion events to be modelled in the derived sessions table.

> **Note:** Because this is part of the sessions table within the Unified dbt package, we still expect your sessions to contain at least one `page_view`, a `page_ping` or a `screen_view` event, and the events must all have a `session_identifier` to be included in the `base_events_this_run_table`. Without a `session_identifier` the event will not be visible to the model, and without a `page_view`, a `page_ping` or a `screen_view` in the session there will be no session record for the model to attach the conversions to.

### Conversion Columns

For every type of conversion you provide a configuration for, the package will add up to 6 columns to the `snowplow_unified_sessions` table, depending on what you provide in your configuration. All columns start with `cv_{name}_` based on the name of your conversion and will be placed at the end of the table.

| Column                       | Description                                                                                  |
| ---------------------------- | -------------------------------------------------------------------------------------------- |
| `cv_{name}_volume`           | An integer column with the total number of conversions in the session                        |
| `cv_{name}_values`           | An array of values for each conversion, if provided in the configuration                     |
| `cv_{name}_total`            | The total of the values for each conversion, if provided in the configuration                |
| `cv_{name}_events`           | An array of `event_id`s for each conversion, if `list_events` is `true` in the configuration |
| `cv_{name}_first_conversion` | The `derived_tstamp` of the first conversion event                                           |
| `cv_{name}_converted`        | A boolean for if there was a conversion in the session                                       |

> **Tip:** If you only want to take into account the first conversion in a session then use the `_converted` column and the first element of the `_values` array in any downstream analysis.

Finally, if `snowplow__total_all_conversions` is set to `true` two additional columns will be added that total the volume and value across all conversions.

| Column           | Description                                  |
| ---------------- | -------------------------------------------- |
| `cv__all_volume` | The sum of volume across all conversions     |
| `cv__all_total`  | The sum of total value across all conversion |

> **Tip:** If you are using these columns make sure that all your conversion values are in the same units e.g. $ or stock volume.

### The conversion configuration dictionary

The `snowplow__conversion_events` variable in our project takes a list of dictionaries to determine what events count as a conversion. These dictionaries are expected to have certain keys and form what we call a conversion configuration. The keys are:

| Key             | Required | Description                                                                                                                                                         | Example value                                              |
| --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `name`          | Yes      | The name of the conversion, to be used in the column names for this conversion                                                                                      | `sign_up`                                                  |
| `condition`     | Yes      | The (valid sql) condition that identifies if the event is a conversion. Must return true or false, and will be used in a `case when ...` statement.                 | `event_name = 'page_view' and page_url like '%signed_up%'` |
| `value`         | No       | The field name or sql to select the value associated with the conversion. If not provided the `cv_{name}_value` and `cv_{name}_total` fields will not be generated. | `tr_total_base/100`                                        |
| `default_value` | No       | The default value to use when a conversion is identified but the value returned is `null`. The type should match value. Default `0`                                 | `1`                                                        |
| `list_events`   | No       | A boolean to determine whether to add the `cv_{name}_events` column to the output.                                                                                  | `true`                                                     |

> **Tip:** Because the sessions table builds from our `_events_this_run` table, and both the `condition` and `value` fields will accept any valid sql that can go in a `case when...` and `select` block respectively, this means you can use any fields in a `contexts_` or `unstruct_` column in addition to the default `atomic.events` fields. However, it will not accept dbt code so you need to extract the relevant field yourself. You can see an example of this below.
>
> For Redshift and Postgres we have added the variable `snowplow__entities_or_sdes` that you can use to bring in all the fields from any self describing events or entities. For more details on this check out the [Custom entities & Events section](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/#custom-entities--events).

#### Example configurations

**All ids of page views on a particular url**

```yml
vars:

  snowplow_unified:

    snowplow__conversion_events:

      [

        {

          "name": "contact_page_view",

          "condition": "event_name = 'page_view' and page_url like '%contact-us%",

          "list_events": true

        }

      ]
```

**Using a self describing event name**

For some self-describing event with a name of `sign_up`, where we do not want to attribute a value:

```yml
vars:

  snowplow_unified:

    snowplow__conversion_events:

      [

        {

          "name": "transact",

          "condition": "event_name = 'sign_up'",

        }

      ]
```

**Using a self-describing event and a context name**

Using our [Snowplow ecommerce tracking](/docs/sources/web-trackers/tracking-events/ecommerce/):

**Snowflake:**

```yml
vars:

  snowplow_unified:

    snowplow__conversion_events:

      [

        {

          "name": "transact",

          "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1:type::varchar = 'transaction'",

          "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0]:revenue::decimal(22,2)",

          "default_value":0

        }

      ]
```

**BigQuery:**

```yml
vars:

  snowplow_unified:

    snowplow__conversion_events:

      [

        {

          "name": "transact",

          "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1_0_0.type = 'transaction'",

          "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1_0_0[SAFE_OFFSET(0)].revenue",

          "default_value":0

        }

      ]
```

**Databricks:**

```yml
vars:

  snowplow_unified:

    snowplow__conversion_events:

      [

        {

          "name": "transact",

          "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1.type = 'transaction'",

          "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0].revenue",

          "default_value":0

        }

      ]
```

**Redshift:**

```yml
vars:

  snowplow_unified:

    snowplow__entities_or_sdes:

      [

        {

          'schema': 'com_snowplowanalytics_snowplow_ecommerce_transaction_1',

          'prefix': 'trans_entity',

          'alias': 'tr',

          'single_entity': true

        },

        {

          'schema': 'com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1',

          'prefix': 'trans_event',

          'alias': 'trev',

          'single_entity': true

        }

      ]

    snowplow__conversion_events:

      [

        {

          "name": "transact",

          "condition": "event_name = 'snowplow_ecommerce_action' and trans_event_type = 'transaction'",

          "value":"trans_entity_revenue"

        }

      ]
```

***

**Defining multiple conversions**

When defining multiple conversion types just follow the (list of dictionaries) format:

```yml
    snowplow__conversion_events:

      [

        {

          "name": "...",

          "condition": "...",

          ...(add more definitions optionally)

        },

        {

          "name": "...",

          "condition": "...",

          ...(add more definitions optionally)

        }

      ]
```

### Configuration Generator

You can use the below generator to generate the conversion events variable, make sure you combine this with existing conversions if you have any already. You can also use the full config generator on the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/unified/) to help you generate all your package variables.

#### Project Variable:

```yaml
vars:

  snowplow_unified:

    snowplow__conversion_events: null
```

## 2. Enabling conversions in the Conversions Module

If you need a conversions source table for your downstream model (e.g. for the snowplow-attribution package), you can enable the optional Conversions module by setting the `snowplow__enable_conversions` variable to `true` to model that for you. You will also need to make sure you define your conversion events using the `snowplow__conversion_events` variable in case you have not already done so for the sessions table.

This would produce an incremental conversions table where you will see the most important fields related to your conversion events based on your definitions. If you defined multiple conversion types, you will see them all in one table.

| event\_id | session\_identifier | user\_identifier     | user\_id | cv\_id | cv\_value | cv\_tstamp              | dvce\_created\_tstamp   | cv\_type     |
| --------- | ------------------- | -------------------- | -------- | ------ | --------- | ----------------------- | ----------------------- | ------------ |
| event\_1  | session\_1          | f000170187170673177  | user\_1  | cv\_1  | 50.00     | 2023-06-08 20:18:32.000 | 2023-06-08 20:18:32.000 | transactions |
| event\_2  | session\_2          | f0009028775170427694 | user\_2  | cv\_2  | 20.42     | 2023-06-11 15:33:03.000 | 2023-06-11 15:33:03.000 | transactions |
| event\_3  | session\_3          | f0008284662789123943 | user\_3  | cv\_3  | 200.00    | 2023-07-07 13:05:55.000 | 2023-07-07 13:05:55.000 | transactions |

> **Note:** On Databricks targets, this table should also have a `cv_tstamp_date` column; this is used as the partition key for the table and should be the `DATE()` of the `cv_tstamp` value.

You can also apply user-stitching to your conversions table by setting the `snowplow__conversion_stitching` variable to `true`. In this case a new field called `stitched_user_id` will be added to the table. For more details please refer to our guide on [Identity Stitching](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/).

---

# Core web vitals module for Unified Digital
> Track Core Web Vitals metrics including LCP, FID, CLS, INP, and TTFB with the Core Web Vitals module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/

This custom module is built as an extension of the dbt-snowplow-unified package, it transforms raw `web vitals` event data into derived tables for easier querying. These events are generated by the **Snowplow Web Vitals plugin** (@snowplow/browser-plugin-web-vitals) of the [JavaScript tracker](/docs/sources/web-trackers/).

To enable this optional module, the unified package must be correctly configured. Please refer to the [quickstart guides](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) for a full breakdown of how to set it up.

## Prerequisites

In order to use this module you would need to:

1. Track core web vitals using the web `Snowplow Web Vitals plugin`, which populates the column/table `unstruct_event_com_snowplowanalytics_snowplow_web_vitals_1`
2. Have the [yauaa enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enabled (for device type and device class), which populates `contexts_nl_basjes_yauaa_context_1`
3. (Ideally, but not necessarily) Have the [spiders and bots](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) enrichment enabled, which populates `contexts_com_iab_snowplow_spiders_and_robots_1`

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw web vitals events:

- `snowplow_unified_web_vitals`: Incremental table used as a base for storing core web vital events (first event per page view).
- `snowplow_unified_web_vital_measurements`: Drop and recompute table to use for visualizations that takes core web vital measurements at the user specified percentile point (defaulted to 75).

## Operation

It is assumed that the Snowplow unified package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_cwv` variable in your `dbt_project.yml` file:

```yaml


vars:

snowplow_unified:

  snowplow__enable_cwv: true

  
```

### Override the module specific macros

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

- The `core_web_vital_page_groups()` [(source)](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/core_web_vital_page_groups.sql) macro is used to let the user classify their urls to specific page groups. It returns the sql to provide the classification (expected in the form of case when statements).

- The `core_web_vital_results_query()` [(source)](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/core_web_vital_results_query.sql) macro is used to let the user classify the thresholds to be applied for the measurements. It returns the sql to provide the logic for the evaluation based on user defined thresholds (expected in the form of case when statements).

Please make sure you set the results you would like the measurements to pass to **`good`** or align it with the `macro_core_web_vital_pass_query()` macro.

- The `core_web_vital_pass_query()` [(source)](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/core_web_vital_pass_query.sql)

```sql
case when lcp_result = 'good' and fid_result = 'good' and cls_result = 'good' then 1 else 0 end passed
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_unified` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_unified` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Snowplow Unified Digital dbt package
> Transform raw web and mobile event data into derived tables for views, sessions, users, and conversions with the Unified Digital dbt package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-unified repo](https://github.com/snowplow/dbt-snowplow-unified), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-unified/#!/overview/snowplow_unified).**

The package contains a fully incremental model that transforms raw web and mobile event data generated by the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) and the [Snowplow iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) or [android trackers](/docs/sources/mobile-trackers/previous-versions/android-tracker/) into a series of derived tables of varying levels of aggregation.

The Snowplow Unified Digital Model aggregates Snowplow's out of the box page view, screen view, and page ping events to create a set of derived tables - views, sessions and users - that contain many useful dimensions as well as calculated measures such as time engaged and scroll depth.

![Unified Digital Model data flow](/assets/images/unified-process-light.drawio-a3ee55188faf2cad375b15273928e694.png)

## Overview

This model consists of a series of modules, each producing a table which serves as the input to the next module. The 'standard' modules are:

- Base: Performs the incremental logic, outputting the table `snowplow_unified_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model. From this table it also generates the `snowplow_unified_events_this_run` table, which coalesces multiple fields to combine web and mobile fields coming from different contexts / sdes.
- Views: Aggregates event level data to page view / screen view level, `view_id`, outputting the table `snowplow_unified_views`.
- Sessions: Aggregates event level data to a session level, `session_identifier`, outputting the table `snowplow_unified_sessions`. Includes other events but requires at least one `page_view`, `screen_view` or `page_ping` event in the session.
- Users: Aggregates session level data to a users level, `user_identifier`, outputting the table `snowplow_unified_users`.
- User Mapping: Provides a mapping between user identifiers, `user_identifier` and `user_id`, outputting the table `snowplow_unified_user_mapping`. This can be used for session stitching.

### Supported Entities

While using any entity in our packages is possible thanks to [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/), a large set of common web and mobile entities are built into the processing of the package to add to your derived tables.

| Entity                                                                                 | Type   | Enabled via Variable                                                                                                                                            |
| -------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [YAUAA](/docs/events/ootb-data/device-and-browser/#yauaa-user-agent-parsing)           | web    | `snowplow__enable_yauaa`                                                                                                                                        |
| [IAB](/docs/events/ootb-data/device-and-browser/#spiders-and-robots)                   | web    | `snowplow__enable_iab`                                                                                                                                          |
| [UA](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)           | web    | `snowplow__enable_ua`                                                                                                                                           |
| [Browser](/docs/events/ootb-data/device-and-browser/#browser-entity)                   | web    | `snowplow__enable_browser_context`, `snowplow__enable_browser_context_2` (depending on schema versions tracked, when both are enabled the values are coalesced) |
| [Mobile](/docs/events/ootb-data/device-and-browser/#mobile-entity)                     | mobile | `snowplow__enable_mobile_context`                                                                                                                               |
| [Geolocation](/docs/events/ootb-data/geolocation/#geolocation-entity)                  | mobile | `snowplow__enable_geolocation_context`                                                                                                                          |
| [Application](/docs/events/ootb-data/app-information/#entity-definitions)              | mobile | `snowplow__enable_application_context`                                                                                                                          |
| [Screen](/docs/events/ootb-data/page-and-screen-view-events/#screen-entity)            | mobile | `snowplow__enable_screen_context`                                                                                                                               |
| [Deep links](/docs/events/ootb-data/links-and-referrers/#deep-links-for-mobile)        | mobile | `snowplow__enable_deep_link_context`                                                                                                                            |
| [Screen summary](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) | mobile | `snowplow__enable_screen_summary_context`                                                                                                                       |

### Optional Modules

| Module                                                                                                                            | Enabled via Variable           |
| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| [Consent reporting](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/)       | `snowplow__enable_consent`     |
| [Core web vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/) | `snowplow__enable_cwv`         |
| [App errors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/app-errors-module/)           | `snowplow__enable_app_errors`  |
| [Conversions](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/)                | `snowplow__enable_conversions` |

## Engaged vs. Absolute Time

At a view- and session-level we provide two measures of time; **absolute**, how long a user had the page open, and **engaged**, how much of that time the user was on the page. Engaged time is often a large predictor of a customer conversion, such as a purchase or a sign-up, whatever that may be in your domain.

Calculating absolute time is simple, it's the difference between the `derived_tstamp` of the first and last (page view or page ping) events within that page view/session.

### Web Calculation

The calculation for engaged time on web is more complicated, it is derived based on page pings which means if the user isn't active on your content, the engaged time does not increase. Let's consider a single page view example of reading an article; partway through the reader may see something they don't understand, so they open a new tab and look this up. They might stumble upon a Wikipedia page on it, they go down a rabbit hole and 10 minutes later they make it back to your site to finish the article. In this case there will be a gap for those 10 minutes in the page pings in the events data.

To adjust for these gaps we calculate engaged time as the time to trigger each ping (your heartbeat) times the number of pings (ignoring the first one), and add to that the time delay to the first ping (your minimum visit length). The formula is:

$$ engaged\_time=t\_{heartbeat}\times (n\_{distinct\_pings} -1) + t\_{min\_visit\_length} $$

and the below shows an example visually for a single page view.

![Page views and pings showing gaps to highlight the difference between absolute and engaged time](/assets/images/engaged_time_light.drawio-7db13a561e8e35da0c9ec1a0d605a344.png)

At a session level, this calculation is slightly more involved, as it needs to happen per page view and account for stray page pings (see below), but the underlying idea is the same.

### Mobile Calculation

For Mobile we use the `screen_summary` entity from the [mobile trackers](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking) for the engaged time. Check out the [mobile engagement demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/) for a live view of this.

## Stray Page Pings (Web only)

Stray Page Pings are pings within a session that do not have a corresponding `page_view` event within **the same session**. The most common cause of these is someone returning to a tab after their session has timed out but not refreshing the page. The `page_view` event exists in some other session, but there is no guarantee that both these sessions will be processed in the same run, which could lead to different results. Depending on your site content and user behavior the prevalence of sessions with stray page pings could vary greatly. For example with long-form content we have seen around 10% of all sessions contain only stray page pings (i.e. no `page_view` events).

We take different approaches to adjust for these stray pings at the page view and sessions levels, which can lead to differences between the two tables, but each is as accurate as we can currently make it.

### Sessions

As all our processing ensures full sessions are reprocessed, our sessions level table includes all stray page ping events, as well as all other view and ping events. We adjust the start time down based on your minimum visit length if the session starts with a page ping, and we include sessions that contain only (stray) pings. We also count page views based on the number of unique `page_view_ids` you have (from the `web_page` context) rather than using absolute `page_view` events to include these stray pings, and account for stray pings in the engaged time. Overall this is a more accurate view of a session and treats the stray pings as if they had a corresponding `page_view` event in the same session, even when they did not.

The result of this is you may see misalignment between sessions and if you tried to recalculate them based directly off the page views table; this is because we discard stray pings during page view processing as discussed below, so the values (`views`, `engaged_time_in_s`, and `absolute_time_in_s`) in the sessions table may be higher, but are more accurate at a session level.

![Stray page ping sessionisation](/assets/images/stray_sessions_light.drawio-fa6d6267e82193eda817072a25e62d82.png)

### Page Views

For page views, because we cannot guarantee the sessions with the `page_view` event and all subsequent `page_ping` events are processed within the same run, we choose to discard all stray page pings. Without doing this it could be possible that you would get different results from different run configurations.

**Without enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_old-light.drawio-99737733b79738a26736f5958f49625f.png)

**With enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_new-light.drawio-2cc5a036f12bec0cdc82533701523a75.png)



> **Info:** Currently we do not process these discarded stray page pings in any way, meaning that engaged time and scroll depth in these cases may be under representative of the true value. Due to session level reprocessing this remains a complicated issue to resolve, but please [let us know](https://github.com/snowplow/dbt-snowplow-unified/issues) if you would like to help solve this!

---

# How to override macros in the Unified Digital package
> Override macros in the Unified Digital dbt package to customize channel classification, conversion definitions, and event processing.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/overridable-macros/

The Unified Digital package includes several macros you can override to customize the package behavior for your specific requirements. This page documents the available macros and their purposes.

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

## Unify fields query

The `unify_fields_query` macro is used to populate the `snowplow_unified_events_this_run` table; groups together fields that come from multiple entities or SDEs depending on whether the platform is mobile or web. Should be overwritten if there is other grouping or complex SQL you wish to run on the `events this run` table.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/unify_fields_query.sql).

## Filter bots

The `filter_bots` macro defines the filter to remove bot events from events processed by the package. The filter is of the form `and <condition>` and is used throughout the package to filter out bots from all models.

The macro takes an optional `table_alias` argument, which is the table alias to prefix the column name with, if any. Default `none`.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/filter_bots.sql).

## Channel group query

The `channel_group_query` macro defines the channel a user arrived at using various fields and populates the `default_channel_group` field in the `views` and `sessions` tables. Must be a valid SQL `select` object, such as a complete `case when` statement. Used as part of the `platform_independent_fields` macro.

The defaults can be altered by overriding the macro in your project to generate your expected channels. Default values will not consider any custom marketing parameters you may have. Your override will most likely be a long list of case statements where `mkt_source` and `mkt_medium` fields are used for the classification. You can also rely on the `source_category` field that you will get as the package automatically joins the seed file generated table `snowplow_unified_dim_ga4_source_categories` where this macro gets used.

If you enable the `snowplow__use_refr_if_mkt_null` variable, the `src_field` will become a `coalesce(mkt_source, refr_source)`, and it will become `coalesce(mkt_medium, refr_medium)` for the `medium_field` used as a reference throughout the original definition.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/channel_group_query.sql).

## Engaged session

The `engaged_session` macro defines whether a session was engaged and populates the `is_engaged` field. Must return `true` or `false` and be a valid SQL `select` object, such as a complete `case when` statement. Used in the `sessions_this_run` table.

The default definition considers a session engaged if it has at least 2 views, or at least 2 heartbeats of engaged time, or had a conversion event.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/engaged_session.sql).

## Content group query

The `content_group_query` macro defines the content groups by classifying the page URLs for views. Must be a valid SQL `select` object, such as a complete `case when` statement. Used in the `views_this_run` table.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/content_group_query.sql).

---

# Snowplow Utils dbt package
> Core macros and helper functions for Snowplow dbt packages including incremental logic, optimized upserts, and utility macros.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/

> **Info:** The models, functionality, and variables described below are only available from `snowplow-utils v0.15.0` and above, as earlier packages do not utilize these variables.

**The package source code can be found in the [snowplow/dbt-snowplow-utils repo](https://github.com/snowplow/dbt-snowplow-utils).**

The package contains a handful of useful macros that can help speed up development, as well as some handy macros to rebuild the `base` level tables that each Snowplow dbt package uses internally. We highly recommend looking at the package [readme](https://github.com/snowplow/dbt-snowplow-utils?tab=readme-ov-file#macros) for a list of all macros in the package that may be of use in any [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) you may build.

> **Info:** This `base` functionality and the process flow described below is only available from `snowplow-utils v0.15.0` and above.

The process flow of these macros and the tables generated is shown below:

![Utils Package data flow](/assets/images/utils-process-light.drawio-aa8ea9da878b2ebfd2b7ec86a91cb75a.png)

You can utilize these macros to create customized versions of the `events_this_run` tables. The pre-built Snowplow dbt packages can then incorporate these tables in addition to your own models. This approach offers several benefits:

Firstly, it allows you to generate custom logic that can be leveraged by the standard dbt packages. This means you can take advantage of the existing functionality while incorporating your own specialized requirements.

Secondly, it enables you to develop custom logic on top of the incremental framework already provided by Snowplow. This ensures that you only process the events that are necessary for your use case, without having to handle the incremental logic yourself. The framework takes care of sessionizing the data, simplifying your workflow.

## Overview

This model consists of a series of macros that generate models, together culminating in the `base_events_this_run` table. This table is built in other projects, including the other Snowplow packages, which can then be leveraged to get insights into your Snowplow data. The models that are built in these other projects are:

- `snowplow_base_quarantined_sessions.sql`: This model contains a list of all quarantined sessions, where a session is quarantined if it's total duration exceeds the maximum specified in the `snowplow__max_session_days` variable which has a default value of 3.
- `snowplow_incremental_manifest.sql`: This model is a manifest which contains a list of all Snowplow related models that are enabled, and keeps track of the last timestamp of successfully parsed data for each model.
- `snowplow_base_new_event_limits.sql`: This model keeps track of the upper and lower limit of events that are going to be considered, which is referenced downstream in other models to limit table scans.
- `snowplow_base_sessions_lifecycle_manifest.sql`: This model maintains a session lifecycle manifest, outlining for each session (based on the specified `session_identifier` and `user_identifier`) the start and end timestamp of the session
- `snowplow_base_sessions_this_run.sql`: This model identifies which sessions are relevant to be processed for the current run, and picks only those sessions from the session lifecycle manifest outlined above.
- `snowplow_base_events_this_run.sql`: This model finally takes the output of the base sessions table and extracts all relevant events from the sessions identified in the base sessions table, along with any custom contexts specified. This table can then be leveraged down the line by other Snowplow dbt packages or by a user's custom models.

With these macros, you can specify which columns get used as timestamps for incremental data processing, and which identifiers are used on a `session` and `user` level. By default, the `session_tstamp` is the `collector_tstamp`, the `session_identifier` is `domain_sessionid`, and the `user_identifier` is `domain_userid`. For a more in-depth explanation into how you can customize these values, you can read the [Utils quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/utils/#6-setting-up-the-sessions-lifecycle-manifest-macro) docs.

---

# Snowplow dbt packages
> Overview of Snowplow dbt packages including Unified Digital, Attribution, Ecommerce, Identities, Media Player, Normalize, and Utils.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/

Snowplow provides several dbt packages to help you get value from your data.

| Package name                                                                                                 | Purpose                                                                            |
| ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)   | Marketing attribution analysis                                                     |
| [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/)   | Transform raw web and mobile event data into derived tables                        |
| [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) | Transform raw media player event data into derived tables                          |
| [Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/)       | Transform raw ecommerce event data into derived tables                             |
| [Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/)       | Normalize events, filtered events, and users table for use in downstream ETL tools |
| [Utils](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/)               | Useful macros                                                                      |
| [Identities](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-identities-data-model/)     | Transform identity entities and merge events into resolved identity tables         |

---

# Snowplow Fractribution dbt package
> Legacy Fractribution dbt package for marketing attribution analysis. Superseded by the Attribution package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-fractribution repo](https://github.com/snowplow/dbt-snowplow-fractribution), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-fractribution/#/overview/snowplow_fractribution).** ​

> **Note:** The Fractribution Package is no longer maintained, please refer to the [Attribution package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) for marketing attribution analysis with Snowplow

The Fractribution package is dependent on the `snowplow_web_page_views` model created by the [snowplow\_web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/) dbt package. Run [snowplow\_web](https://github.com/snowplow/dbt-snowplow-web) if you do not have data in the `snowplow_web_page_views` table for the period of time you will run fractribution for.

## Overview

​ The Snowplow Fractribution dbt package is a tool to perform Attribution Modeling. The name itself comes from `Fractional attribution`, which allows you to attribute the value of a conversion to one or more channels depending on the conversion pathway. As a result, it becomes possible to determine the revenue per channel, as well as ROAS (Return On Ad Spend - the amount of revenue that is earned for every dollar spent on advertising) once you have cost data for each marketing channel. ​

> **Tip:** While the throughout this package we use the term _channel_, it is entirely possible to define this in terms of something else e.g. a campaign, and define the spend and classification in the `channel_spend` and `channel_classification` macros respectively.

This package consists of a series of dbt models that produce the following tables:

- `snowplow_fractribution_channel_counts`: Number of sessions grouped by channel, campaign, source and medium
- `snowplow_fractribution_channel_spend`: Spend on each channel, used in ROAS calculations
- `snowplow_fractribution_conversions_by_customer_id`: Conversion revenue for each conversion, along with the associated `customerid`
- `snowplow_fractribution_path_summary`: Summary of different path combinations and associated conversion/non-conversions
- `snowplow_fractribution_paths_to_conversion`: Path combinations leading to conversion
- `snowplow_fractribution_paths_to_non_conversion`: Path combinations leading to non-conversion
- `snowplow_fractribution_sessions_by_customer_id`: Channel information by session timestamp, where an event timestamp is considered as the session start

Once the models are generated, the next step is to run a python script which is included in the package to run the attribution calculations. You can run this locally, via a docker image, or using Snowpark if you are on Snowflake. In the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) section you will find a step-by-step guide on how to operate the package as a whole.

This script will generate and populate the following three additional tables:

- `snowplow_fractribution_channel_attribution`: The main output table that shows conversions, revenue, spend and ROAS per channel
- `snowplow_fractribution_path_summary_with_channels`: The conversion and revenue attribution per channel (used to create the report table)
- `snowplow_fractribution_report_table`: An intermediate table that shows, for each unique path, a summary of conversions, non conversions and revenue, as well as which channels were assigned a contribution ​

### Intra-session Channels

​ In Google Analytics (Universal Analytics) a new session is started if a campaign source changes (referrer of campaign tagged URL). Snowplow utilizes activity based sessionization, rather than campaign based sessionization. Setting `consider_intrasession_channels` to `false` will take only the campaign information from the first page view in a given Snowplow session, and not give credit to other channels in the converting session if they occur after the initial page view.

### Filter unwanted / wanted channels

You can specify a list of channels for the variable `snowplow__channels_to_exclude` to exclude them from analysis (if kept empty all channels are kept). For example, users may want to exclude the 'Direct' channel from the analysis.

You can also do the opposite, filter on certain channels to include in your analysis. You can do so by specifying them in the list captured within the variable `snowplow__channels_to_include`. ​

### Path Transform Options

​ Paths to conversion are often similar, but not identical. As such, path transforms reduce unnecessary complexity in similar paths before running the attribution algorithm. ​

1. `exposure (default)`: the same events in succession are reduced to one: `A → A → B` becomes `A → B`, a compromise between first and unique
2. `unique`: all events in a path are treated as unique (no reduction of complexity). Best for smaller datasets (small lookback window) without a lot of retargeting
3. `first`: keep only the first occurrence of any event: `A → B → A` becomes `A → B`, best for brand awareness marketing
4. `frequency`: keep a count of the events’ frequency: `A → A → B` becomes `A(2) → B`, best when there is a lot of retargeting
5. `remove_if_last_and_not_all`: requires a channel to be added as a parameter, which gets removed from the latest paths unless it removes the whole path as it is trying to reach a non-matching channel parameter: E.g target element: `A` path: `A → B → A → A` becomes `A → B`
6. `remove_if_not_all`: requires a channel to be added as a parameter, which gets removed from the path altogether unless it would result in the whole path's removal: E.g target element: `A` path: `A → B → A → A` becomes `B` ​

### Attribution Models

The package currently offers 5 different attribution models, that can be chosen by setting the `attribution_model` flag when calling the python script.

- `shapley` (default): Takes the weighted average of the marginal contributions of each channel to a conversion
- `first_touch`: Assigns 100% attribution to the first channel in each path.
- `last_touch`: Assigns 100% attribution to the last channel in each path.
- `position_based`: The first and last channels get 40% of the credit each, with the remaining channels getting the leftover 20% distributed evenly.
- `linear`: Assigns attribution evenly between all channels on the path. ​

![Data processing model for the fractibution package](/assets/images/attribution_models_light-6421813fd5263308643dee86dc1f23e9.png)

### Package Macros

> **Tip:** To overwrite these macros correctly with those in your project, ensure you prefix the macro name by `default__` in the definition e.g.
>
> ```jinja2
> {% macro default__conversion_value() %}
>
>     tr_total/100
>
> {% endmacro %}
> ```

#### `conversion_clause` macro

> The [`conversion_clause`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_clause.sql) macro specifies how to filter Snowplow events to only succesfful conversion events. How this is filtered will depend on your definition of a conversion. The default is filtering to events where `tr_total > 0`, but this could instead filter on `event_name = 'checkout'`, for example. If you are using the ecommerce model, you will still need to set this for the fractribution code to run (even though _all_ events are conversions in the ecommerce model), in this case change it to `transaction_revenue > 0`.

#### `conversion_value` macro

> The [`conversion_value`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_value.sql) macro specifies either a single column or a calculated value that represents the value associated with that conversion. The default is `tr_total`, but revenue or a calculation using revenue and discount\_amount from the default ecommerce schema, for example, could similarly be used.

#### `channel_classification` macro

> The [`channel_classification`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_classification.sql) macro is used to perform channel classifications. This can be altered to generate your expected channels if they differ from the channels generated in the default macro. It is highly recommended that you examine and configure this macro when using your own data, as the ROAS calculations and attribution calculations will run against these channel definitions, and the default values will not consider any custom marketing parameters.

#### `channel_spend` macro

> The [`channel_spend`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_spend.sql) macro is used to query the spend by channels. It requires a user supplied SQL script to extract the total ad spend by channel.
>
> Required output format:
>
> - `channel`: STRING NOT NULL
> - `spend`: FLOAT64 _(or other warehouse equivalent)_ (Use the same monetary units as conversion revenue, and NULL if unknown.)

### Environment Variables

For the script or docker container to run you need to set some environment variables that can be accessed to connect to your warehouse. These variables vary by warehouse. If you are using Docker, it can be easier to set these in a file that you reference when you run the container. See the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) for detailed running information.

**BigQuery:**

- `project_id`: Project id of your BigQuery warehouse
- `bigquery_dataset`: Dataset for your derived tables For the python script only (for Docker you mount this as a volume at run time):
- `google_application_credentials`: Google Service Account JSON file

**Databricks:**

- `databricks_schema`: Schema for your derived tables
- `databricks_server_hostname`: Databricks server hostname
- `databricks_http_path`: Databricks compute resources URL
- `databricks_token`: Personal Access Token

**Snowflake:**

- `snowflake_account`: Snowflake account ID
- `snowflake_user`: Snowflake username
- `snowflake_password`: Snowflake password
- `snowflake_user_role`: Snowflake role
- `snowflake_warehouse`: Snowflake warehouse
- `snowflake_database`: Snowflake database
- `snowflake_schema`: Schema for your derived tables

**Redshift:**

- `redshift_host`: Redshift host url
- `redshift_database`: Redshift database
- `redshift_port`: Redshift port (likely `5439`)
- `redshift_user`: Redshift user
- `redshift_password`: Redshift password
- `redshift_schema`: Redshift schema

***

All warehouses require the following if you are using the docker container:

- `conversion_window_start_date`: The start date of your conversion window, in the format of YYYY-MM-DD
- `conversion_window_end_date`: The end date of your conversion window, in the format of YYYY-MM-DD
- `attribution_model`: The [attribution model method](#attribution-models)
- `warehouse`: The warehouse to run on, one of `snowflake`, `bigquery`, `redshift`, or `databricks`

​

### Differences to Google's Fractribution

​ There are some changes from [Google's](https://github.com/google/fractribution) Fractribution code that have been noted below. ​

- Temporary UDFs have been converted to persistent / permanent UDFs
- Some temporary tables converted to permanent tables
- Users without a user\_id are treated as 'anonymous' ('f') users and the domain\_userid is used to identify these sessions
- Users with a user\_id are treated as identified ('u') users
- Templating is now run almost entirely within dbt rather than the custom SQL / Jinja templating in the original Fractribution project
- Channel changes and contributions within a session can be considered using the `consider_intrasession_channels` variable

---

# App errors in Mobile package
> App errors module for the legacy Mobile dbt package. Superseded by the Unified Digital App errors module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/app-errors-module/

Exception tracking captures any unhandled exceptions within the application. The exception tracking is enabled by default, and it allows the tracker to intercept critical exceptions in the app. Exceptions can crash the app so it's likely that the event will be sent after the restart of the app. Being a critical situation we can't be 100% sure that all the exception stacktraces are reliably stored for sending before the crash of the app.

## App Error Columns

| Column                | Description                                                                 |
| --------------------- | --------------------------------------------------------------------------- |
| `programmingLanguage` | The name of the programming language used in which the app error occured.   |
| `message`             | The error message that the application showed when the app error occurred.  |
| `threadName`          | The name of the process that ran the thread when the app error occurred.    |
| `threadId`            | The ID of the thread in which the app error occurred.                       |
| `stackTrace`          | The full stack trace that was presented when the app error occured.         |
| `causeStackTrace`     | The cause of the stack trace that was presented when the app error occured. |
| `lineNumber`          | The line number in the code where the app error occured.                    |
| `className`           | The name of the class where the app error occurred.                         |
| `exceptionName`       | The name of the exception encountered in the app error.                     |
| `isFatal`             | A boolean to describe whether the app error was fatal or not.               |
| `lineColumn`          | The line number in the code where the app error occured.                    |
| `fileName`            | The name of the file where the app error occurred.                          |

---

# Snowplow Mobile dbt package
> Legacy Mobile dbt package for mobile event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

**The package source code can be found in the [snowplow/dbt-snowplow-mobile repo](https://github.com/snowplow/dbt-snowplow-mobile), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-mobile/#!/overview/snowplow_mobile).**

The package contains a fully incremental model that transforms raw mobile event data generated by the [Snowplow iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) or [android trackers](/docs/sources/mobile-trackers/previous-versions/android-tracker/) into a series of derived tables of varying levels of aggregation.

The Snowplow mobile data model aggregates Snowplow's out-of-the-box mobile events to create a set of derived tables - screen views, sessions, and users. These contain many useful dimensions, as well as calculated measures such as screen views per session.

![Mobile Package data flow](/assets/images/mobile-process-light.drawio-de145671692376e99ef55abe866f03dc.png)

| snowplow-mobile version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres |
| ----------------------- | ------------------ | -------- | ---------- | -------- | --------- | -------- |
| 1.0.0                   | >=1.6.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.7.2                   | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.6.3                   | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.5.5                   | >=1.0.0 to <1.3.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.2.0                   | >=0.20.0 to <1.0.0 | ✅        | ❌          | ✅        | ✅         | ✅        |

## Overview

This model consists of a series of modules, each producing a table which serves as the input to the next module. The 'standard' modules are:

- Base: Performs the incremental logic, outputting the table `snowplow_mobile_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model.
- Screen Views: Aggregates event level data to a screen view level, on `screen_view_id`, outputting the table `snowplow_mobile_screen_views`.
- Sessions: Aggregates screen view level data to a session level, on `session_id`, outputting the table `snowplow_mobile_sessions`.
- Users: Aggregates session level data to a users level, on `device_user_id`, outputting the table `snowplow_mobile_users`.
- User Mapping: Provides a mapping between user identifiers, `device_user_id` and `user_id`, outputting the table `snowplow_mobile_user_mapping`. This can be used for session stitching.

## Contexts

The following contexts can be enabled depending on your tracker configuration:

- Mobile context
- Geolocation context
- Application context
- Screen context

By default they are disabled. They can be enabled by configuring the `dbt_project.yml` file and setting the appropriate `snowplow__enable_{context_type}_context` variable to `true`.

## Optional Modules

Currently the App Errors module, used for crash reporting, is the only optional module. More will be added in the future as the tracker's functionality expands.

Assuming your tracker is capturing `application_error` events, the module can be enabled by configuring the `dbt_project.yml` file:

```yaml


vars:

snowplow_mobile:

  snowplow__enable_app_errors_module: true

  
```

---

# Consent module for the Web package
> Consent module for the legacy Web dbt package. Superseded by the Unified Digital Consent module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/consent-module/

This custom module is built as an extension of the dbt-snowplow-web package.

It transforms raw `consent_preferences` and `cmp_visible` event data into derived tables for easier querying. These events are generated by the **Enhanced Consent plugin** of the [JavaScript tracker](/docs/sources/web-trackers/).

> **Info:** For the incremental logic to work within the module you **must** use at least `RDB Loader v4.0.0`, as the custom module relies on the additional `load_tstamp` field for dbt native incrementalisation.
>
> Whenever a new consent version is added to be tracked, the model expects an `allow_all` event in order to attribute the events to the full list of latest consent scopes. It is advisable to send a test event of that kind straight after deployment so that the model can process the data accurately.

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw consent tracking events:

- `snowplow_web_consent_log`: Snowplow incremental table showing the audit trail of consent and Consent Management Platform (cmp) events.
- `snowplow_web_consent_users`: Incremental table of user consent tracking stats.
- `snowplow_web_consent_totals`: Summary of the latest consent status, per consent version.
- `snowplow_web_cconsent_scope_status`: Aggregate of current number of users consented to each consent scope.
- `snowplow_web_cmp_stats`: Used for modeling cmp\_visible events and related metrics.
- `snowplow_web_consent_versions`: Incremental table used to keep track of each consent version and its validity.

## Operation

It is assumed that the Snowplow web package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_consent` variable most conveniently set in your dbt\_project.yml file:

```yaml


vars:

snowplow_web:

  snowplow__enable_consent: true

  
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_web` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_web` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Modeling conversions with the Web package
> Conversion modeling for the legacy Web dbt package. Superseded by the Unified Digital Conversions module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/conversions/

Conversion events are a type of event that's important to your business, be that a transaction, a sign up to a newsletter, or the view of a specific page. Whatever type of event matters to you, so long as it can be determined from a single event record, you'll be able to model and aggregate these conversions at a session level with our package.

Because we know that each user may have a different concept of what a conversion means to them, for example it could be a specific type of event such as a `page_view` on a specific url, a self-describing event such as `sign_up`, or even a bespoke `conversion` event type with some attached context, the way we model conversions is also flexible and allows you to change your definition over time. This allows you to track multiple types of conversions, with varying logic, even retrospectively.

> **Note:** Because this is part of the sessions table within the web package, we still expect your sessions to contain at least one `page_view` or `page_ping` event, and the events must all have a `domain_sessionid` to be included in the `base_events_this_run_table`. Without a `domain_sessionid` the event will not be visible to the model, and without a `page_view` or `page_ping` in the session there will be no session record for the model to attach the conversions to.

## Conversion Columns

For every type of conversion you provide a configuration for, the package will add up to 6 columns to the `snowplow_web_sessions` table, depending on what you provide in your configuration. All columns start with `cv_{name}_` based on the name of your conversion and will be placed at the end of the table.

| Column                       | Description                                                                                  |
| ---------------------------- | -------------------------------------------------------------------------------------------- |
| `cv_{name}_volume`           | An integer column with the total number of conversions in the session                        |
| `cv_{name}_values`           | An array of values for each conversion, if provided in the configuration                     |
| `cv_{name}_total`            | The total of the values for each conversion, if provided in the configuration                |
| `cv_{name}_events`           | An array of `event_id`s for each conversion, if `list_events` is `true` in the configuration |
| `cv_{name}_first_conversion` | The `derived_tstamp` of the first conversion event                                           |
| `cv_{name}_converted`        | A boolean for if there was a conversion in the session                                       |

> **Tip:** If you only want to take into account the first conversion in a session then use the `_converted` column and the first element of the `_values` array in any downstream analysis.

Finally, if `snowplow__total_all_conversions` is set to `true` two additional columns will be added that total the volume and value across all conversions.

| Column           | Description                                  |
| ---------------- | -------------------------------------------- |
| `cv__all_volume` | The sum of volume across all conversions     |
| `cv__all_total`  | The sum of total value across all conversion |

> **Tip:** If you are using these columns make sure that all your conversion values are in the same units e.g. $ or stock volume.

## The conversion configuration dictionary

The `snowplow__conversion_events` variable in our project takes a list of dictionaries to determine what events count as a conversion. These dictionaries are expected to have certain keys and form what we call a conversion configuration. The keys are:

| Key             | Required | Description                                                                                                                                                         | Example value                                              |
| --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `name`          | Yes      | The name of the conversion, to be used in the column names for this conversion                                                                                      | `sign_up`                                                  |
| `condition`     | Yes      | The (valid sql) condition that identifies if the event is a conversion. Must return true or false, and will be used in a `case when ...` statement.                 | `event_name = 'page_view' and page_url like '%signed_up%'` |
| `value`         | No       | The field name or sql to select the value associated with the conversion. If not provided the `cv_{name}_value` and `cv_{name}_total` fields will not be generated. | `tr_total_base/100`                                        |
| `default_value` | No       | The default value to use when a conversion is identified but the value returned is `null`. The type should match value. Default `0`                                 | `1`                                                        |
| `list_events`   | No       | A boolean to determine whether to add the `cv_{name}_events` column to the output.                                                                                  | `true`                                                     |

> **Tip:** Because the sessions table builds from our `_events_this_run` table, and both the `condition` and `value` fields will accept any valid sql that can go in a `case when...` and `select` block respectively, this means you can use any fields in a `contexts_` or `unstruct_` column in addition to the default `atomic.events` fields. However, it will not accept dbt code so you need to extract the relevant field yourself. You can see an example of this below.
>
> For Redshift and Postgres users currently you are limited to just the fields added by the IAB, UA, and YAUAA contexts without modifying the models yourself, however we plan to support adding arbitrary self-describing event and context fields to our base table in the future.

### Example configurations

**All ids of page views on a particular url**

```json
    {

    "name": "contact_page_view",

    "condition": "event_name = 'page_view' and page_url like '%contact-us%",

    "list_events": true

    }
```

**Using a self describing event name**

For some self-describing event with a name of `sign_up`, where we do not want to attribute a value:

```json
    {

    "name": "transact",

    "condition": "event_name = 'sign_up'",

    }
```

**Using a self-describing event and a context name**

Using our [Snowplow ecommerce tracking](/docs/sources/web-trackers/tracking-events/ecommerce/):

**Snowflake:**

```json
    {

    "name": "transact",

    "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1:type::varchar = 'transaction'",

    "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0]:revenue::decimal(22,2)",

    "default_value":0

    }
```

**BigQuery:**

```json
    {

    "name": "transact",

    "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1_0_0.type = 'transaction'",

    "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1_0_0[SAFE_OFFSET(0)].revenue",

    "default_value":0

    }
```

**Databricks:**

```json
    {

    "name": "transact",

    "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1.type = 'transaction'",

    "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0].revenue",

    "default_value":0

    }
```

***

## Configuration Generator

You can use the full config generator on the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/web/) to help you generate your conversion event definitions.

---

# Core web vitals module for the Web package
> Core Web Vitals module for the legacy Web dbt package. Superseded by the Unified Digital Core Web Vitals module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/core-web-vitals-module/

This custom module is built as an extension of the dbt-snowplow-web package, it transforms raw `web vitals` event data into derived tables for easier querying. These events are generated by the **Snowplow Web Vitals plugin** (@snowplow/browser-plugin-web-vitals) of the [JavaScript tracker](/docs/sources/web-trackers/).

To enable this optional module, the web package must be correctly configured. Please refer to the [quickstart guides](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) for a full breakdown of how to set it up.

## Prerequisites

In order to use this module you would need to:

1. Track core web vitals using the web `Snowplow Web Vitals plugin`, which populates the column/table `unstruct_event_com_snowplowanalytics_snowplow_web_vitals_1`
2. Have the [yauaa enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enabled (for device type and device class), which populates `contexts_nl_basjes_yauaa_context_1`
3. (Ideally, but not necessarily) Have the [spiders and bots](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) enrichment enabled, which populates `contexts_com_iab_snowplow_spiders_and_robots_1`

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw web vitals events:

- `snowplow_web_web_vitals`: Incremental table used as a base for storing core web vital events (first event per page view).
- `snowplow_web_web_vital_measurements`: Drop and recompute table to use for visualizations that takes core web vital measurements at the user specified percentile point (defaulted to 75).

## Operation

It is assumed that the Snowplow web package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_cwv` variable in your `dbt_project.yml` file:

```yaml


vars:

snowplow_web:

  snowplow__enable_cwv: true

  
```

### Override the module specific macros

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

- The `core_web_vital_page_groups()` [(source)](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/core_web_vital_page_groups.sql) macro is used to let the user classify their urls to specific page groups. It returns the sql to provide the classification (expected in the form of case when statements).

- The `core_web_vital_results_query()` [(source)](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/core_web_vital_results_query.sql) macro is used to let the user classify the thresholds to be applied for the measurements. It returns the sql to provide the logic for the evaluation based on user defined thresholds (expected in the form of case when statements).

Please make sure you set the results you would like the measurements to pass to **`good`** or align it with the `macro_core_web_vital_pass_query()` macro.

- The `core_web_vital_pass_query()` [(source)](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/core_web_vital_pass_query.sql)

```sql
case when lcp_result = 'good' and fid_result = 'good' and cls_result = 'good' then 1 else 0 end passed
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_web` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_web` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Snowplow Web dbt package
> Legacy Web dbt package for web event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

**The package source code can be found in the [snowplow/dbt-snowplow-web repo](https://github.com/snowplow/dbt-snowplow-web), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-web/#!/overview/snowplow_web).**

The package contains a fully incremental model that transforms raw web event data generated by the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) into a series of derived tables of varying levels of aggregation.

The Snowplow web data model aggregates Snowplow's out of the box page view and page ping events to create a set of derived tables - page views, sessions and users - that contain many useful dimensions as well as calculated measures such as time engaged and scroll depth.

![Web Package data flow](/assets/images/web-process-light.drawio-a2b276daa0d6a441c3851bc3bb5db941.png)

| snowplow-web version | dbt versions        | BigQuery | Databricks | Redshift | Snowflake | Postgres |
| -------------------- | ------------------- | -------- | ---------- | -------- | --------- | -------- |
| 1.0.1                | >=1.6.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.16.2               | >=1.5.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.15.2               | >=1.4.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅\*      |
| 0.13.3\*\*           | >=1.3.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.11.0               | >=1.0.0 to <1.3.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.5.1                | >=0.20.0 to <1.0.0  | ✅        | ❌          | ✅        | ✅         | ✅        |
| 0.4.1                | >=0.18.0 to <0.20.0 | ✅        | ❌          | ✅        | ✅         | ❌        |

\* Since version 0.15.0 of `snowplow_web` at least version 15.0 of Postgres is required, otherwise you will need to [overwrite](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/) the `default_channel_group` macro to not use the `regexp_like` function.

\*\* From version v0.13.0 onwards we use the `load_tstamp` field so you must be using [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v4.0.0 and above, or [BigQuery Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v1.0.0 and above. If you do not have this field because you are not using these versions, or you are using the Postgres loader, you will need to set `snowplow__enable_load_tstamp` to `false` in your `dbt_project.yml` and will not be able to use the consent models.

## Overview

This model consists of a series of modules, each producing a table which serves as the input to the next module. The 'standard' modules are:

- Base: Performs the incremental logic, outputting the table `snowplow_web_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model.
- Page Views: Aggregates event level data to a page view level, `page_view_id`, outputting the table `snowplow_web_page_views`.
- Sessions: Aggregates event level data to a session level, `domain_sessionid`, outputting the table `snowplow_web_sessions`. Includes other events but requires at least one `page_view` or `page_ping` event in the session.
- Users: Aggregates session level data to a users level, `domain_userid`, outputting the table `snowplow_web_users`.
- User Mapping: Provides a mapping between user identifiers, `domain_userid` and `user_id`, outputting the table `snowplow_web_user_mapping`. This can be used for session stitching.

## Optional Modules

See the subsections for information on the optional modules.

## Overridable Macros

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

- `filter_bots(table_alias)`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/filter_bots.sql): used to define the filter to remove bot events from events processed by the package. Of the form `and <condition>`. Used throughout the package to filter out bots from all models.
- `channel_group_query()`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/channel_group_query.sql): defines the channel a user arrived at using various fields, populates the `default_channel_group` field. Must be a valid sql `select` object e.g. a complete `case when` statement. Used in `sessions_this_run` table.
- `engaged_session()`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/engaged_session.sql): defines if a session was engaged or not, populates the `is_engaged` field. Must return `true` or `false` and be a valid sql `select` object e.g. a complete `case when` statement. Used in `sessions_this_run` table.
- `content_group_query()`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/channel_group_query.sql): defines the content groups by classifying the page urls for page views. Must be a valid sql `select` object e.g. a complete `case when` statement. Used in `page_views_this_run` table.

## Engaged vs. Absolute Time

At a page view- and session-level we provide two measures of time; **absolute**, how long a user had the page open, and **engaged**, how much of that time the user was on the page. Engaged time is often a large predictor of a customer conversion, such as a purchase or a sign-up, whatever that may be in your domain.

Calculating absolute time is simple, it's the difference between the `derived_tstamp` of the first and last (page view or page ping) events within that page view/session.

The calculation for engaged time is more complicated, it is derived based on page pings which means if the user isn't active on your content, the engaged time does not increase. Let's consider a single page view example of reading an article; partway through the reader may see something they don't understand, so they open a new tab and look this up. They might stumble upon a Wikipedia page on it, they go down a rabbit hole and 10 minutes later they make it back to your site to finish the article. In this case there will be a gap for those 10 minutes in the page pings in the events data.

To adjust for these gaps we calculate engaged time as the time to trigger each ping (your heartbeat) times the number of pings (ignoring the first one), and add to that the time delay to the first ping (your minimum visit length). The formula is:

$$ engaged\_time=t\_{heartbeat}\times (n\_{distinct\_pings} -1) + t\_{min\_visit\_length} $$

and the below shows an example visually for a single page view.

![Page views and pings showing gaps to highlight the difference between absolute and engaged time](/assets/images/engaged_time_light.drawio-7db13a561e8e35da0c9ec1a0d605a344.png)

At a session level, this calculation is slightly more involved, as it needs to happen per page view and account for [stray page pings](#stray-page-pings), but the underlying idea is the same.

## Stray Page Pings

Stray Page Pings are pings within a session that do not have a corresponding `page_view` event within **the same session**. The most common cause of these is someone returning to a tab after their session has timed out but not refreshing the page. The `page_view` event exists in some other session, but there is no guarantee that both these sessions will be processed in the same run, which could lead to different results. Depending on your site content and user behavior the prevalence of sessions with stray page pings could vary greatly. For example with long-form content we have seen around 10% of all sessions contain only stray page pings (i.e. no `page_view` events).

We take different approaches to adjust for these stray pings at the page view and sessions levels, which can lead to differences between the two tables, but each is as accurate as we can currently make it.

### Sessions

As all our processing ensures full sessions are reprocessed, our sessions level table includes all stray page ping events, as well as all other view and ping events. We adjust the start time down based on your minimum visit length if the session starts with a page ping, and we include sessions that contain only (stray) pings. We also count page views based on the number of unique `page_view_ids` you have (from the `web_page` context) rather than using absolute `page_view` events to include these stray pings, and account for stray pings in the engaged time. Overall this is a more accurate view of a session and treats the stray pings as if they had a corresponding `page_view` event in the same session, even when they did not.

The result of this is you may see misalignment between sessions and if you tried to recalculate them based directly off the page views table; this is because we discard stray pings during page view processing as discussed below, so the values (`page_views`, `engaged_time_in_s`, and `absolute_time_in_s`) in the sessions table may be higher, but are more accurate at a session level.

![Stray page ping sessionisation](/assets/images/stray_sessions_light.drawio-fa6d6267e82193eda817072a25e62d82.png)

### Page Views

For page views, because we cannot guarantee the sessions with the `page_view` event and all subsequent `page_ping` events are processed within the same run, we choose to discard all stray page pings. Without doing this it could be possible that you would get different results from different run configurations.

**Without enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_old-light.drawio-99737733b79738a26736f5958f49625f.png)

**With enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_new-light.drawio-2cc5a036f12bec0cdc82533701523a75.png)



> **Info:** Currently we do not process these discarded stray page pings in any way, meaning that engaged time and scroll depth in these cases may be under representative of the true value. Due to session level reprocessing this remains a complicated issue to resolve, but please [let us know](https://github.com/snowplow/dbt-snowplow-web/issues) if you would like to help solve this!

---

# Overview of legacy dbt models
> Legacy Snowplow dbt packages including Web, Mobile, and Fractribution models superseded by newer packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/

> **Note:** Legacy packages are no longer under active development and are in some cases no longer supported. Please look for a suitable replacement for these packages where possible.

| Package name  | Purpose                                             | Superseded by                                                                                              |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Web           | Transform raw web event data into derived tables    | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Mobile        | Transform raw mobile event data into derived tables | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Fractribution | Marketing attribution analysis                      | [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) |

---

# Run modules asynchronously
> Run dbt package modules asynchronously at different intervals using session identifiers and selective model execution.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/async-runs/

You may wish to run the modules asynchronously, for instance run the views module hourly but the sessions and users modules daily. You would assume this could be achieved using e.g.:

```bash
dbt run --select +snowplow_web.page_views
```

However, due to the models being listed in the [manifest table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) we would need to know which models are included in this at run time; this is not currently possible. Instead all models from the standard and custom modules are selected from the manifest table and the package will attempt to synchronize all models. This makes the above command unsuitable for asynchronous runs.

However you can leverage dbt's `ls` command in conjunction with shell substitution to explicitly state what models to run, allowing a subset of models to be selected from the manifest and thus run independently.

For example to run just the page views module asynchronously:

```bash
dbt run --select +snowplow_unified.views --vars "{'models_to_run': '$(dbt ls --select +snowplow_unified.views --output name | tail -n +4)'}"
```

> **Note:** The `| tail -n +4` simply removes the dbt logging front matter from the output

---

# Backfill your data
> Steps for backfilling the data models after initialization, adding custom models, or following a full refresh.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/

When you first start using our packages, when you add [new custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/), or following on from a [full refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/), you may have a large period of data that needs to be processed before your models are _caught up_ with your live data and the package does a [standard run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-4-standard-run).

As described in [incremental processing](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) by default our packages are set up to avoid processing large periods of data in one go, this helps to reduce the workload on your cloud, as well as reducing the impact of any issues in the data causing a single run to fail.

When backfilling your data, it is recommended to increase the `snowplow__backfill_limit_days` variable to as high as you feel comfortable with to reduce the number of runs that need to complete to get your processed data up to date. In general increasing this to cover 3-6 months is a good balance between speed and number of runs.

We also recommend doing this process manually, as until your data is up to date, no new data will be processed, so it may take many runs for your data to catch up with live tracking. Running locally you can do many runs back-to-back rather than wait for your scheduled runs to take place. You may also wish to warn any users of your data this is planned as it can lead to delays, and an initial full-refresh will remove any existing data from the derived tables.

> **Tip:** If you have added a new model it will be processed from the value in your `snowplow__start_date` variable, if you don't need the model to start from this far back then you should change this for the first run so the new model only runs from a more recent date instead.

## Blue-Green style Deployment

As the process of backfilling data may take a long time depending on your data volumes, and during this time no new data will be processed, it may be preferable to backfill your model(s) in a staging environment and then deploy these derived tables into your production schema at a later date.

> **Warning:** This process involves manually editing the manifest table, which is dangerous and should be done only with understanding of our [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) and [manifest tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/).

Let's assume you have one schema named `prod` that is used by a regularly scheduled run of a project that uses our package(s), and you want to introduce a new custom model backfilled with all your historic Snowplow data. We are going to first process this model in a `staging` schema before migrating it across.

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    %% This comment is here to trigger a git change when spaces at the end of lines are removed, do not do that on this page it is required for the axis formatting
    axisFormat
    tickInterval 1day
    section New Model
    Unprocessed :crit, n1, 2020-01-01, 5d
    section Other Models
    Processed :active, a1, 2020-01-01, 5d
```

### Step 1: Create and process new model only

In a local environment, assuming you are using version control for your project, create a new branch and create your model in this branch. Next ensure you are using a profile/target and/or configure the package models so that they will write to the `staging` schema (or be prefixed with `staging` depending on your setup) instead of `prod`.

Now you will need to run the below script to run only what is needed for your new model (see [model selection](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) for more details on why). We've also increased the backfill limit days so this needs to be run fewer times.

```bash
dbt run --select +my_new_model --vars "{'models_to_run': '$(dbt ls --select  +my_new_model --output name | tail -n +4)', 'snowplow__backfill_limit_days': 90}"
```

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 1day
    section New Model
    Unprocessed :crit, n1, after n2, 4d
    Processed (in staging) :active, n2, 2020-01-01, 1d
    section Other Models
    Processed :active, a1, 2020-01-01, 5d
```

Once you have run this enough times that the data is up-to-date, which you can identify either by looking at the print out in the logs or by querying the manifest table, it's time to move this data to your `prod` schema.

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 1day
    section New Model
    Processed (in staging) :active, n2, 2020-01-01, 5d
    section Other Models
    Processed :active, a1, 2020-01-01, 5d
```

### Step 2: Migrate the new table

The exact SQL you need to run will vary by warehouse, and some warehouses may offer the ability to do this in their UI. It may be possible to move your table via an `ALTER TABLE` statement, however for simplicity you can recreate your new table in your `prod` schema via a `CREATE TABLE` statement e.g.

```sql
CREATE TABLE prod.my_new_model as (select * from staging.my_new_model);
```

Make sure to recreate any additional features as necessary e.g. partition keys.

### Step 3: Update the manifest record

> **Warning:** Ensure your project is not running while you perform this step as it could cause a conflict in the manifest table.

If you were to now add this model to the project being run in `prod`, because the manifest table in `prod` (or more likely, `prod_snowplow_manifest`) has no record of this model, it will process it again from the start (as it will be in [state 2](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-2-new-model-introduced)). To avoid this you need to add a record for this model into the manifest table.

Again, some UIs may allow you to edit a table, but an `INSERT` statement is recommended e.g.

```sql
INSERT INTO PROD.SNOWPLOW_<package>_INCREMENTAL_MANIFEST (MODEL, LAST_SUCCESS) VALUES

('my_new_model', timestamp '2023-10-11 10:11:12');
```

Where you can find the exact timestamp you need to use in your `staging` manifest table.

### Step 4: Migrate the model

Finally, with the data in the `prod` schema, and the record of it in the manifest table, you can merge your branch into `main` (or whatever branch you run your `prod` from), ensuring that the schema setting for your model matches. On the next run the package will pick up this new model, see it in the manifest, and run it alongside the rest of the models without needing to start from scratch.

---

# Tests for dbt packages
> Use dbt tests to validate data quality in Snowplow packages with test selection tags for scratch and derived models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/dbt-tests/

[Tests](https://docs.getdbt.com/docs/build/tests) are a useful feature in dbt to ensure that your data meets some expectations of it, such as certain fields should not be null. We provide a suite of tests with our packages to help test the output tables for any issues, however you could and should add your own as no one knows the required quality checks of your data better than you.

## Test Selection

The packages contain tests for both the scratch and derived models. Depending on your use case you might not want to run all tests in production, for example to save costs. There are several tags included in the packages to help select subsets of tests. Tags:

- `this_run`: Any model with the `_this_run` suffix
- `scratch`: Any model in the scratch sub directories.
- `derived`: Any of the derived models i.e. page views, sessions and users.
- `primary-key`: Any test on the primary keys of all models in this package.

For example if your derived tables are very large you may want to run the full test suite on the `this_run` tables, which act as the input for the derived tables, but only primary key schema tests on the derived tables to ensure no duplicates. If using such a set up, we would also recommend including the `page/screen_view_in_session_value` data test for the page/screen views derived tables.

This is our recommended approach to testing and can be implemented using the selector flag (see [YAML selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section for more details) as follows:

```bash
dbt test --selector snowplow_<package>_lean_tests
```

This is equivalent to:

```bash
dbt test --select snowplow_<package>,tag:this_run # Full tests on _this_run models

dbt test --select snowplow_<package>,tag:manifest # Full tests on manifest models

dbt test --select snowplow_<package>,tag:primary-key,tag:derived # Primary key tests only on derived tables.

dbt test --select snowplow_<package>,tag:derived,test_type:data  # Include the page/screen_view_in_session_value data test
```

Alternatively, if you wanted to run all available tests both in the Snowplow package, and your custom modules:

```bash
dbt test --selector snowplow_<package>
```

## Test Configuration

There are a number of configuration options on dbt tests, which are documented in the [dbt docs](https://docs.getdbt.com/reference/test-configs). You may want to apply some of these on the tests within the Snowplow dbt packages to ensure they're applicable to your data and use cases. The best way to achieve this is by editing the configuration within your project's root `dbt_project.yml` file.

For example, you may want to apply a severity threshold on all tests within the `snowplow_web` package. You can do this by updating your `dbt_project.yml` configuration as below:

```yml
# dbt_project.yml

# Example config applied to all tests in a package

...

tests:

  snowplow_web:

    +warn_if: ">10"

    +error_if: ">100"
```

Alternatively, you could apply a configuration to a specific, single instance of a test by calling out its resource path directly:

```yml
# dbt_project.yml

# Example config applied to a specific test

...

tests:

  snowplow_web:

    page_views:

      not_null_snowplow_web_page_views_domain_sessionid:

        +where: "app_id != '<app_id_to_exclude_from_test>'"
```

---

# Debug data modeling issues
> Guide for debugging data modeling issues including execution errors, missing data, and unexpected results.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/debugging/

There will inevitably be times when you run into dbt errors or the output is not what you expected (e.g. a session is missing from the sessions table). This guide will try and help you get started with the most common issues and how to handle them, to help you get unblocked faster and avoid having to open a customer ticket to ask for help.

## Scenario 1. The model errors out during execution

### What is the specific error message?

The first thing to look for is the actual error message. Depending on how you run your package you might have different levels of access: e.g. running the dbt locally will give you access to the whole model run log and compiled sql code within the `target` folder, whereas in Console you will typically only be able to see the model's run log with the error message where it fails.

If you look at the run log, you may need to scroll down the output until you reach the error though (there may be more than one). You will also see which models ran successfully. Look for the full error message, if it starts with `Snowplow Error: ...` it is an error message specifically created by the Snowplow Package you run, and it should give you a context into where and why it is failing. It could also be a warehouse related issue, due to access rights, or temporary resource issues etc.

In case of a Snowplow Console based error message, an error output message such as a `connection refused` related message, or for example an `EOF` error is typically down to some external database process, or a network connection error. These errors often do not need any interaction to be resolved.

If it is a syntax issue, it may give a specific code fragment to help you debug. Note down which model fails and try to execute that particular model in debug mode locally by adding `-d` to your dbt run prompt. Dbt should display a more extensive log message potentially with the full compiled code it tries to run but fails that you can copy and take to your sql editor of choice to debug separately.

### Is this the first time running into this issue? Did the model run properly before?

If the error happens after the initial package setup, it may be related to specific configurations missing. This is especially true in case of the Unified package due to its extensive optional feature support. We added a specific config helper variable: `snowplow__enable_initial_checks` to help with errors resulting from wrong configurations. When enabled it should give you a more specific error log with respect to your config that might help in such cases.

If the model only ran once but not the second time, the issue is most likely related to the incrementalization, in many cases the second run of the model contains extra code related to the timeframe to take into account for incremental updates.

If the package was running fine several times, but all of a sudden stopped working, you may need to check what changes happened since the last run. Did you upgrade to a new dbt version? Can it be related to a snowplow package version change? Was there something implemented from the tracking side? Were there any recent pipeline issues?

## Scenario 2. Output is missing data

The most common scenario that may happen is there are certain session\_identifiers or user\_identifiers missing from the output tables. The reason behind this can be many, we will touch the most common ones.

We process the data based on sessions: for each run we take a look at the new events (with default 6 hours of lookback window) then look at the sessions manifest table and identify how far to look back in time to cover those sessions with new events as a whole. One overlooked common scenario is that in case of pipeline issues, if some data is getting processed by the model, but the rest are only added to the warehouse outside of that default 6 hour lookback window, then the package will leave those events behind unless the same session will have new events coming in, and it needs to be reprocessed as a whole again (unless the data modeling jobs were immediately paused after the pipeline went partially down). In such cases you need partial backfilling (see the [late loaded events](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/#late-loaded-events) section).

The events can also be [late sent](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/#late-sent-events), in which case if they fall out of the limit, they will not get processed.

We also "quarantine" the sessions at times, as however hard we may try to filter out bots, there could be odd outliers that keep sessions alive for days, for which we have the `snowplow__max_session_days` variable. If bots keep sending events for longer than defined in that variable, the package will only process the first of those in a subsequent run and archives the session\_identifier in the [quarantine table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#quarantine-table) after it runs. In such a case it is always best to look for the missing event's session\_identifier (domain\_userid as defaulted for web events) in the table to explain why the event is missing.

Other root causes may be missing timestamp fields (e.g. `dvce_created_tstamp`, `dvce_sent_tstamp` that are needed for the sessionization logic).

If the issue is not related to the session not being processed, but certain fields appear in the views table but not downstream in a more aggregated layer (e.g. the users table) it may simply be due to the nature of the aggregation. There are certain rules to which event per session the packages take a certain field (e.g. first based on collector\_tstamp or derived\_tstamp but only if it is a page\_view event etc.).

To help debug it better, it is best to choose a sample session, select all its events from the events table as a whole, and see based on the compiled model code (when executing dbt locally, you can find it in the target folder) how the joins work to figure out why certain things are not working the way there are "expected". There may be changes that need to take place in tracking or through a custom data model for it to work in a specific use case.

One recommendation when deep-diving into errors is to look at dbt's lineage graph to see the model evolution, and work your way upstream to find out where the data gets filtered out using the compiled sql code from the given run. Once you find the first source when the data starts appearing you know which is the model where it gets filtered out, and you can check the subsequent model's compiled code, then cte by cte you can execute the code. It should not impact the model as those will be select statements, and you will most likely be able to identify the root cause.

> **Info:** You can inspect the lineage graph to verify this if you are unsure by using dbt's built in data lineage feature through dbt docs. We also update that for each of our latest releases on github: <https://snowplow.github.io/dbt-snowplow-unified/#!/overview?g_v=1>

---

# Turn off models
> How to turn off a model in our packages using model configuration in your dbt project yaml file.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/disabling-models/

If you do not require certain modules provided by the package you have the option to disable them using the [model configuration](https://docs.getdbt.com/reference/resource-configs/enabled) in your project yaml. For instance to disable the users module in the `snowplow_unified` package:

```yml
models:

  snowplow_unified:

    users:

      enabled: false
```

For other models, you can identify the path to the models in your `dbt_packages` folder in your project. Note you should disable the entire folder, now just the derived model itself otherwise the [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#other-this-run-tables) tables will still process each run.

> **Tip:** Any dependent modules will also need to be disabled - for instance if you disabled the sessions module in the unified package, you will also have to disable the users module.

---

# Full or partial table refreshes
> Perform full or partial refreshes of Snowplow dbt package models to reprocess data or fix issues.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/

You can choose to refresh all or only some tables.

## Complete refresh of Snowplow package

While you can drop and recompute the incremental tables within this package using the standard `--full-refresh` flag, all [manifest tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) are protected from being dropped in production (as defined by a target not matching your `snowplow__dev_target_name`).

Without dropping the manifest during a full refresh, the selected derived incremental tables would be dropped but the processing of events would resume from where the package left off, as captured by the `snowplow_<package_name>_incremental_manifest` table, rather than your `snowplow__start_date`.

In order to drop all the manifest tables and start again set the `snowplow__allow_refresh` var to `true` at run time:

**Other packages:**

```bash
dbt run --select snowplow_<package_name> tag:snowplow_<package>_incremental --full-refresh --vars 'snowplow__allow_refresh: true'

# or using selector flag

dbt run --selector snowplow_<package_name> --full-refresh --vars 'snowplow__allow_refresh: true'
```

**Attribution:**

```bash
dbt run --full-refresh --vars 'snowplow__allow_refresh: true'
```

***

When doing a full refresh of the package, it will begin again from your `snowplow__start_date` and backfill based on the calculations explained in the [Incremental Sessionization Logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) page. Please ensure you trigger enough runs to catch back up with live data, or adjust your variables for these runs accordingly (see our page on [backfilling](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/) for more information).

## Refresh only some models in a package

You may at times wish to refresh only some derived models in a package, either a built in one, or a [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/). This may be due to changes you have made to variable, a change in logic, or some other reason. To achieve this:

1. Manually drop the table(s) in your database (you may wish to simply rename them until the back-fill is completed in case of any issues).

2. Remove the models from the manifest table; this can be achieved either by:

   1. _(Recommended)_ using the `models_to_remove` variable at run time

   ```bash
   dbt run --select +snowplow_<package_name>_model_name --vars '{snowplow__start_date: "yyyy-mm-dd", models_to_remove: snowplow_<package>_model_name}'
   ```

   2. _(High Risk)_ manually deleting the record from the `snowplow_<package_name>_incremental_manifest` table.

By removing the model from the manifest the `<package\>` will be in [State 2](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-2-new-model-introduced) and will replay all events.

## Custom period re-processing

There may be times where you need to re-run some or all of the models for a period of data, potentially due to incorrect supplemental data brought into a model, or in case of a mistake in your model code itself. There are two methods to do this: one is simpler but must complete in a single run, the other is more complex but can be executed over multiple runs. Either will work for any custom models, assuming they were built using [Snowplow incremental materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/).

Note that it is not currently possible to re-process a fixed period in the past, it is only possible to re-process from a given date up to the current date of the data.

> **Note:** Both methods are only suitable if the value of your source data in your `upsert` and date keys have not changed (e.g. `view_id` and `derived_tstamp` for the `snowplow_unified_views` model). If this is not the case, or your custom models are not built using this approach, there is no choice but to run a full refresh of the model.

### Option 1: Altering the look back window

As defined in the [standard run incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-4-standard-run), the lower limit for the events to be processed in the models is `max_last_success - snowplow__lookback_window_hours`. If you increase the value of `snowplow__lookback_window_hours` to a number that goes beyond the period you wish to re-run from, then all events from that time will be reprocessed and fed through all models in the package.

For example, if your last run success was `2022-10-30 13:00:00` and you needed to reprocess events from `2022-10-25 02:00:00`, you would set your `snowplow__lookback_window_hours` to `137` (5 days × 24 hours + 11 hours, and 6 hours of an additional buffer the look back window would usually provide). This will reprocess all the events in a single run, which may be larger than the value you have set in `snowplow__backfill_limit_days`. If you wish to avoid going over the backfill limit you have set, please use option 2. Don't forget to change your value back once the run has completed!

![Demonstration of data processing through option 1 approach](/assets/images/data_progress_example1_light.drawio-07bc53a2da644ef13f0119dc545560b2.png)

### Option 2: Manipulating the manifest table

It is possible to update the value in the [manifest tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) so that the `max_last_success` values for the relevant models are set to the date you wish to re-run from. In this case, it may take multiple runs for your data to be completely re-run, based on your `snowplow__backfill_limit_days` value, and you may be left with abnormal data until all runs are completed. In the case where you have aggregations in your models (such as the `snowplow_unified_users` model), or have downstream reports/visualization/ML models based on these tables, we recommend setting the `snowplow__backfill_limit_days` to a large enough value that it can be completed in one run to minimize downstream issues (or to use option 1, where this happens by default).

For example, if your last run success was `2022-10-30 13:00:00` and you needed to reprocess events from `2022-10-25 02:00:00`, you would set the value in your manifest table for the model(s) to `2022-10-25 02:00:00`. This will then process data from that point (minus the `snowplow__lookback_window_hours` buffer) until either the current date, or according to your `snowplow__backfill_limit_days`, whichever yields a smaller time period. This will repeat until the data is fully reprocessed.

![Demonstration of data processing through multiple runs of the option 2 approach](/assets/images/data_progress_example2_light.drawio-7391f248be421c70a9aefa793669b5bc.png)

> **Note:** Manipulating the values in the manifest tables can cause unexpected outcomes if you don't understand the Snowplow [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/)). Where possible, use option 1.

---

# Operate Snowplow dbt packages
> How to operate our dbt packages including backfilling, model selection, debugging, and asynchronous runs.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/

There are many ways to operate and run our packages, including backfilling or a-sync running of specific tables. The following pages contain guides for how to do this.

## Required Privileges

In addition to the standard [privileges required by dbt](https://docs.getdbt.com/faqs/warehouse/database-privileges), our packages by default write to additional schemas beyond just your profile schema. If your connected user does not have `create schema` privileges, you will need to ensure that the following schemas exist in your warehouse and the user can create tables in them:

- `<profile_schema>_derived`
- `<profile_schema>_scratch`
- `<profile_schema>_snowplow_manifest`

Alternatively, you can override the output schemas our models write to, see the relevant package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for how to do this.

**Snowflake:**

```sql
  grant create schema on database <database_name> to role <role_name>;



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to role <role_name>;

   grant usage on schema <profile_schema>_scratch to role <role_name>;

   grant usage on schema <profile_schema>_manifest to role <role_name>;
```

For more information, please refer to the [Official Guide](https://docs.snowflake.com/en/sql-reference/sql/grant-privilege) on setting up permissions.

**BigQuery:**

Please refer to the [Official Guide](https://cloud.google.com/bigquery/docs/access-control) on setting up permissions.

**Databricks:**

```sql
   -- user with "use catalog" privilege on the catalog

  grant create schema on catalog <catalog_name> to <principal_name>



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to <user_name>;

   grant usage on schema <profile_schema>_scratch to <user_name>;

   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to service principal, or group instead of users), please refer to the [Official Guide](https://docs.databricks.com/en/sql/language-manual/security-grant.html) on setting up permissions.

**Redshift:**

```sql
   -- someone with superuser access

   create schema authorization <user_name>;



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to <user_name>;

   grant usage on schema <profile_schema>_scratch to <user_name>;

   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to role, or group instead of users), please refer to the [Official Guide](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) on setting up permissions.

**Postgres:**

```sql
   -- someone with superuser access

   create schema authorization <user_name>;



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to <user_name>;

   grant usage on schema <profile_schema>_scratch to <user_name>;

   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more information, please refer to the [Official Guide](https://www.postgresql.org/docs/current/sql-createschema.html) on setting up permissions.

***

---

# Run models on data lakes
> How to run our models on lakehouses including Databricks and other data lake platforms.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/lakes/

> **Info:** Running the models on data lakes or lakehouses (using external tables in a warehouse to read directly from a lake) is currently in Early Release state and is not fully supported. Certain features may not work as expected and errors are more likely to occur. Please use this approach at your own risk and raise any issues you find with us.

If you are using the [lake loaders](/docs/api-reference/loaders-storage-targets/lake-loader/) to load your data into a lake storage option, it may be possible to use our data models. In general in this section of the docs we are not going to detail which warehouses support which file formats, or how to set up the respective tables in each warehouse - please see the docs for your appropriate warehouse to see what file formats they support.

## Databricks

At time of writing, `delta` is the preferred file format for Databricks [external tables](https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html). If you create an external table from this lake format in Databricks, you should be able to run the models without any further changes required by simply pointing the model at this table.

## Snowflake

At time of writing, `Iceberg` is the preferred file format for Snowflake [iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg). If you wish to use our models with this, currently only the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) package supports this, by setting the `snowplow__snowflake_lakeloader` variable to `true`.

Note that compared to the other loaders for Snowflake, that field names in Self-describing events and Entities are converted to `snake_case` format (the other loaders retain the format used in the schema, often `camelCase`). You will need to adjust other variables and inputs accordingly compared to what you may find in the docs.

## Spark

At time of writing, `Iceberg` is the supported file format for Spark external tables. We've tested this using Glue and Thrift as a connection method. If you have your event data in Iceberg format in a lake, you should be able to run the models by pointing the packages to a spark deployment, connected to that lake. For more information on setting up dbt with Spark using Thrift, please refer to the [dbt Spark documentation on Thrift](https://docs.getdbt.com/docs/core/connect-data-platform/spark-setup#thrift).

## Redshift (spectrum)

Currently using Redshift Spectrum tables is not supported for our packages due to [limitations](https://docs.aws.amazon.com/redshift/latest/dg/nested-data-restrictions.html) with the platform.

## BigQuery on GCS

Currently using GCS/BigQuery external tables is not tested but may work, please let us know your experience if you try this.

---

# Model selection
> Select and run specific Snowplow dbt models using tags and selection syntax to keep incremental tables in sync.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/

The Snowplow models in each package are designed to be run as a whole, which ensures all incremental tables are kept in sync. As such, run the model using:

```bash
dbt run --select snowplow_<package> tag:snowplow_<package>_incremental
```

The `snowplow_<package>` selection will execute all nodes within the relevant Snowplow package, while the `tag:snowplow_<package>_incremental` will execute all custom modules that you may have created.

Given the verbose nature of this command we suggest using the YAML selectors we have provided. The equivalent command using the selector flag would be:

```bash
dbt run --selector snowplow_<package>
```

Within the packages we have provided a suite of suggested selectors to run and test the models within the packages. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax).

These are defined in each `selectors.yml` file within the packages, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

---

# Attribution Quickstart
> Quick start guide for the Snowplow Attribution dbt package to analyze marketing attribution and conversion paths.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/attribution/

> **Info:** We are gradually moving tutorials like the QuickStart Guides from our standard Documentation to the **Tutorials & Guides** section. You can now find the latest tutorial on how to get started with the Attribution package [here](/tutorials/attribution/intro/).

---

# Ecommerce Quickstart
> Quick start guide for the Snowplow Ecommerce dbt package to model cart, checkout, product, and transaction data.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/ecommerce/

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web events dataset being available in your database, the requirements are:

- A dataset of ecommerce events from the [Snowplow JavaScript tracker](/docs/sources/web-trackers/), or the [iOS/Android](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) trackers must be available in the database.
- Have the [`webPage` context](/docs/sources/web-trackers/tracker-setup/initialization-options/) enabled. (Note if you have only tracked mobile ecommerce events, you will need other events in your warehouse to have used this context as we require the column to exist).
- Have the following ecommerce contexts enabled: `cart`, `checkout_step`, `page` `transaction`, `user`
- Track the ecommerce tracking action events on your website/mobile application

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_ecommerce

  version: 1.0.1
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the ecommerce model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-ecommerce/blob/main/selectors.yml)) within the package, however in order to use these selectors you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile), in the table labeled `events`. In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_ecommerce:

    snowplow__atomic_schema: schema_with_snowplow_events

    snowplow__database: database_with_snowplow_events

    snowplow__events_table: table_of_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Filter your data set

You can specify both `start_date` at which to start processing events, the `app_id`'s to filter for, and the `event_name` value to filter on. By default the `start_date` is set to `2020-01-01`, all `app_id`'s are selected, and only the `snowplow_ecommerce_action` name is being surfaced. To change this please add/modify the following in your `dbt_project.yml` file:

```yml
vars:

  snowplow_ecommerce:

    snowplow__start_date: 'yyyy-mm-dd'

    snowplow__app_id: ['my_app_1','my_app_2']

    snowplow__ecommerce_event_names: ['snowplow_ecommerce_action', 'my_custom_ecommerce_event']
```

### 5. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `load_tstamp` or `collector_tstamp`. Unless it is partitioned on `derived_tstamp` (legacy behavior) you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `snowplow__session_timestamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>
>   snowplow_ecommerce:
>
>     snowplow__derived_tstamp_partitioned: false
> ```

### 6. Removing unused modules

The ecommerce package creates tables that depend on the existence of certain entities that are a part of the [Snowplow ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) JS plugin. If, for some reason, you have not implemented them and would like to streamline your data modeling not to create empty tables, then you need to add that configuration to your `dbt_project.yml` file. Below you can see an example of what that would look like if you wanted to disable the [cart entity](/docs/events/ootb-data/ecommerce-events/#cart)

#### Disabling the cart module in `dbt_project.yml` (recommended)

```yml


...

vars:

    snowplow_ecommerce:

        snowplow__disable_ecommerce_carts: true

...

models:

    snowplow_ecommerce:

        carts:

            +enabled: false
```

Adding these two configurations to your `dbt_project.yml` will ensure that the carts module is disabled.

#### Disabling the cart module using `dbt run`

If you want to temporarily disable a module, or you just find it easier to use the command line, you can also do this in the command line when executing the `dbt run` command. You will need to run the following command to disable the carts module

```bash
dbt run --exclude carts --select snowplow_ecommerce --vars '\{snowplow__disable_ecommerce_carts: true}'
```

### 7. Enable mobile ecommerce events

Mobile ecommerce events may be processed in the package, if they have a `domain_sessionid` and are in your listed `app_id`s, however to correctly source the mobile session and screen view ids you need to set the following in your `dbt_project.yml`:

```yml
vars:

  snowplow_ecommerce:

    snowplow__enable_mobile_events: true
```

### 8. Optimize your project

There are ways how you can deal with [high volume optimizations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/) at a later stage, if needed, but you can do a lot upfront by selecting carefully which variable to use for `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. This timestamp column should ideally be set to the column your event table is partitioned on. It is defaulted to `collector_tstamp` but depending on your loader it can be the `load_tstamp` as the sensible value to use:

```yml
vars:

  snowplow_ecommerce:

    snowplow__session_timestamp: 'load_tstamp'
```

### 9. Verify your variables using our Config guides (Optional)

If you are unsure whether the default values set are good enough in your case or you would already like to maximize the potential of your models, you can dive deeper into the meaning behind our variables on our [Config](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/) page. It includes a [Config Generator](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/#config-generator) to help you create all your variable configurations, if necessary.

### 10. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_ecommerce
```

---

# Identities Quickstart
> Quick start guide for the Snowplow Identities dbt package to transform identity entities and merge events into resolved identity tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/identities/

This guide walks you through setting up the [Snowplow Identities dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-identities-data-model/).

## Requirements

- dbt 1.10.6 or later
- Snowflake or BigQuery warehouse
- The [Snowplow Identities service](/docs/identities/) must be enabled and generating identity entities and merge events in your atomic schema

## Installation

Add the package from [GitHub](https://github.com/snowplow-incubator/dbt-snowplow-identities) to your `packages.yml`:

```yaml
packages:

  - git: "https://github.com/snowplow-incubator/dbt-snowplow-identities.git"

    revision: "main"
```

Then run:

```bash
dbt deps
```

## Configure the package

The steps below walk through each variable you need to set in your `dbt_project.yml` to get the package running. Check out the [configuration reference](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/identities/) for more details.

### 1. Accept the license

The package is licensed under the [Snowplow Personal and Academic License (SPAL)](/personal-and-academic-license-1.0/). You must accept the license before the package will run. Add the following to your `dbt_project.yml`:

```yaml
vars:

  snowplow_identities:

    snowplow__license_accepted: true
```

### 2. Override the dispatch order

To take advantage of the optimized upserts that the Snowplow packages offer, ensure that certain macros are called from `snowplow_utils` before `dbt-core`. Add the following to the top level of your `dbt_project.yml`:

```yaml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

### 3. Set the start date

Set `snowplow__start_date` to the date from which you want to begin processing identity data:

```yaml
vars:

  snowplow_identities:

    snowplow__start_date: 'yyyy-mm-dd'
```

### 4. Configure your identifiers

The `snowplow__identifiers` variable controls which event fields are extracted as identifiers and how they appear in the output tables. Each entry has a `reference` (the column name in your events table) and an `alias` (the name used in the output). By default, `domain_userid` and `user_id` are included:

```yaml
vars:

  snowplow_identities:

    snowplow__identifiers:

      - reference: domain_userid

        alias: domain_userid

      - reference: user_id

        alias: user_id
```

This list should reflect the identifiers you have [configured in the Identities service](/docs/identities/configuration/) via Console. You can also include additional identifiers from your events that you want available in the output tables, even if they aren't configured in Console for identity resolution.

### 5. Check source data

The package reads from your atomic events table. By default, it assumes the `atomic` schema in your `target.database`. To override this, add the following to your `dbt_project.yml`:

```yaml
vars:

  snowplow_identities:

    snowplow__atomic_schema: schema_with_snowplow_events

    snowplow__database: database_with_snowplow_events
```

### 6. Filter your data set _(optional)_

This step is optional. To restrict processing to specific app IDs, set `snowplow__app_id`:

```yaml
vars:

  snowplow_identities:

    snowplow__app_id: ['my_app_1', 'my_app_2']
```

### 7. Hash identifiers _(optional)_

This step is optional. If your identifiers contain PII, set `snowplow__hash_identifiers` to `true`. This applies a SHA-256 hash to all `id_value` entries in the `snowplow_identities_identifier_mapping` output table:

```yaml
vars:

  snowplow_identities:

    snowplow__hash_identifiers: true
```

### 8. Run the package

Run the package with this command:

```bash
dbt run --select snowplow_identities
```

Run this package before any other Snowplow dbt packages, so that downstream models can reference the resolved identities.

## Full refresh

By default, running `dbt run --full-refresh` won't drop the incremental manifest, as this would reset all incremental processing. To allow a full reset, set `snowplow__allow_refresh` to `true` before running:

```yaml
vars:

  snowplow_identities:

    snowplow__allow_refresh: true
```

On development targets, the manifest is always dropped on full refresh without needing this flag. Development targets are identified by the `snowplow__dev_target_name` variable, which you can set to match your development target name if it's not the default `dev`.

---

# Quick start guide for Snowplow dbt packages
> Quick start guides for setting up and running Snowplow dbt packages including database privileges and package installation steps.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/

The following pages contain the steps to get you up and running with our packages. For more detailed information on running the packages see the [individual package pages](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/), and the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) pages.

## Database Privileges

In addition to the standard [privileges required by dbt](https://docs.getdbt.com/faqs/warehouse/database-privileges), our packages by default write to additional schemas beyond just your profile schema. If your connected user does not have `create schema` privileges, you will need to ensure that the following schemas exist in your warehouse and the user can create tables in them:

- `<profile_schema>_derived`
- `<profile_schema>_scratch`
- `<profile_schema>_snowplow_manifest`

Alternatively, you can override the output schemas our models write to, see the relevant package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for how to do this.

**Snowflake:**

```sql
  grant create schema on database <database_name> to role <role_name>;



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to role <role_name>;

   grant usage on schema <profile_schema>_scratch to role <role_name>;

   grant usage on schema <profile_schema>_manifest to role <role_name>;
```

For more information, please refer to the [Official Guide](https://docs.snowflake.com/en/sql-reference/sql/grant-privilege) on setting up permissions.

**BigQuery:**

Please refer to the [Official Guide](https://cloud.google.com/bigquery/docs/access-control) on setting up permissions.

**Databricks:**

```sql
   -- user with "use catalog" privilege on the catalog

  grant create schema on catalog <catalog_name> to <principal_name>



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to <user_name>;

   grant usage on schema <profile_schema>_scratch to <user_name>;

   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to service principal, or group instead of users), please refer to the [Official Guide](https://docs.databricks.com/en/sql/language-manual/security-grant.html) on setting up permissions.

**Redshift:**

```sql
   -- someone with superuser access

   create schema authorization <user_name>;



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to <user_name>;

   grant usage on schema <profile_schema>_scratch to <user_name>;

   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to role, or group instead of users), please refer to the [Official Guide](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) on setting up permissions.

**Postgres:**

```sql
   -- someone with superuser access

   create schema authorization <user_name>;



   --alternatively

   create schema <profile_schema>_derived;

   create schema <profile_schema>_scratch;

   create schema <profile_schema>_manifest;

   grant usage on schema <profile_schema>_derived to <user_name>;

   grant usage on schema <profile_schema>_scratch to <user_name>;

   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more information, please refer to the [Official Guide](https://www.postgresql.org/docs/current/sql-createschema.html) on setting up permissions.

***

## Mixing Variables

> **Tip:** When using multiple dbt packages you must be careful to specify which scope a variable or configuration is defined within. In general, always specify each value in your `dbt_project.yml` nested under the specific package e.g.
>
> ```yml
> vars:
>
>   snowplow_web:
>
>     snowplow__atomic_schema: schema_with_snowplow_web_events
>
>   snowplow_mobile:
>
>     snowplow__atomic_schema: schema_with_snowplow_mobile_events
> ```
>
> You can read more about variable scoping in dbt's docs around [variable precedence](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#variable-precedence).

---

# Fractribution Quickstart
> Quick start guide for the legacy Snowplow Fractribution dbt package for marketing attribution modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/fractribution/

> **Note:** The Fractribution Package is no longer maintained, please refer to the [Attribution package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) for marketing attribution analysis with Snowplow

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web events dataset being available in your database:

- have `snowplow_web_page_views` derived table available as a source (generated by the [snowplow\_web package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/))

- have a table with revenue data by users (`domain_userid`, `user_id`) that serves as another source for the fractribution calculations, you can choose either of the following options:

  - your `atomic.events` table with any [self-describing event](/docs/fundamentals/events/#self-describing-events) that captures revenue data
  - the `snowplow_ecommerce_transaction_interactions` derived table generated by the [snowplow\_ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) package
  - any custom incremental table that is built on top of the `snowplow_web` model that results in an aggregated revenue dataset

- `python` or `docker` installed (or you can use Snowpark on Snowflake)

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_fractribution

  version: 0.3.6
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Set variables

The package has some variables that need to be set before it can be run, you should edit these in your `dbt_project.yml` file. Further customization can be done via the variables listed in the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/fractribution/).

- `snowplow__conversion_window_start_date`: The start date in UTC for the window of conversions to include
- `snowplow__conversion_window_end_date`: The end date in UTC for the window of conversions to include
- `snowplow__conversion_hosts`: `url_hosts` to process
- `snowplow__path_transforms`: A dictionary of path transforms and their arguments (see [Path Transform Options](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/#path-transform-options) section)

```yml
vars:

  snowplow_fractribution:

    snowplow__conversion_window_start_date: '2022-01-01'

    snowplow__conversion_window_end_date: '2023-02-01'

    snowplow__conversion_hosts: ['mysite.com']

    snowplow__path_transforms: \{'exposure_path' : null}
```

> **Tip:** If you are using Snowflake, you can automatically run the python scripts using Snowpark when running the dbt package. This is done using macros that create and run a stored procedure on Snowpark after the dbt models have completed.
>
> To enable this you need to set some additional variables. For example, to enable this and use the `last_touch` attribution model:
>
> ```yml
> vars:
>
>   snowplow_fractribution:
>
>     snowplow__run_python_script_in_snowpark: true
>
>     snowplow__attribution_model_for_snowpark: 'last_touch'
> ```

**Modifying the conversions source**

By default the `snowplow__conversions_source` is your atomic events table. In most cases this is likely to be what you want to use, however you may wish to use the in-built conversions modeling as part of our [web package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/conversions/) if you have already defined this, by setting `snowplow__conversions_source` to `"\\{\\{ ref('snowplow_web_sessions') }}\\}"`.

Alternatively, if you are using Redshift/Postgres you may wish to include additional fields from a Self-Describing Event, or an Entity. To do this, you should create a new model in your project, e.g. `models/snowplow/snowplow_joined_events_table.sql` which should have something like the following content:

> _For more information about dealing with duplicates and the macro in this code, make sure to see our [deduplication docs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/)._

```jinja2
with {{ snowplow_utils.get_sde_or_context('atomic',

                                          'my_custom_context',

                                          "'\{\{ get_lookback_date_limits(\"min\") }}\}'",

                                          "'\{\{ get_lookback_date_limits(\"max\") }}\}'",

                                          'my_prefix')}}



select

  events.*,

  b.*

from \\{\\{ source('atomic', 'events') }}\\} as events

  left join nl_basjes_my_prefix_1 b on

    events.event_id = b.my_prefix__id

    and events.collector_tstamp = b.my_prefix__tstamp



where

  -- use the appropriate partition key to filter on in addition to this, add a bit of a buffer if it is not derived_tstamp

  date(derived_tstamp) >= '\{\{ get_lookback_date_limits(\"min\") }}\}'

  and date(derived_tstamp) <= '\{\{ get_lookback_date_limits(\"max\") }}\}'
```

Finally ensure you set the `snowplow__conversions_source` to `"\\{\\{ ref('snowplow_joined_events_table') }}\\}"`

### 3. Configure macros

> **Tip:** While the macro and matching columns/tables use the term _channel_, it is entirely possible to define this in terms of something else e.g. a campaign.

All the below macros are created with the intention to let users modify them to fit their personal use case. If you wish to change this, copy the macro from the macros folder in the `snowplow_fractribution` package (at `[dbt_project_name]/dbt_packages/snowplow_fractribution/macros/conversion_clause.sql`) and add it to the macros folder of your own dbt project where you are free to make any alterations. You will find a detailed guide / illustration with sample code within the individual macros themselves.

> **Tip:** To overwrite these macros correctly with those in your project, ensure you prefix the macro name by `default__` in the definition e.g.
>
> ```jinja2
> {% macro default__conversion_value() %}
>
>     tr_total/100
>
> {% endmacro %}
> ```

#### `conversion_clause` macro

> The [`conversion_clause`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_clause.sql) macro specifies how to filter Snowplow events to only succesfful conversion events. How this is filtered will depend on your definition of a conversion. The default is filtering to events where `tr_total > 0`, but this could instead filter on `event_name = 'checkout'`, for example. If you are using the ecommerce model, you will still need to set this for the fractribution code to run (even though _all_ events are conversions in the ecommerce model), in this case change it to `transaction_revenue > 0`.

#### `conversion_value` macro

> The [`conversion_value`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_value.sql) macro specifies either a single column or a calculated value that represents the value associated with that conversion. The default is `tr_total`, but revenue or a calculation using revenue and discount\_amount from the default ecommerce schema, for example, could similarly be used.

#### `channel_classification` macro

> The [`channel_classification`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_classification.sql) macro is used to perform channel classifications. This can be altered to generate your expected channels if they differ from the channels generated in the default macro. It is highly recommended that you examine and configure this macro when using your own data, as the ROAS calculations and attribution calculations will run against these channel definitions, and the default values will not consider any custom marketing parameters.

#### `channel_spend` macro

> The [`channel_spend`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_spend.sql) macro is used to query the spend by channels. It requires a user supplied SQL script to extract the total ad spend by channel.
>
> Required output format:
>
> - `channel`: STRING NOT NULL
> - `spend`: FLOAT64 _(or other warehouse equivalent)_ (Use the same monetary units as conversion revenue, and NULL if unknown.)

### 4. Run the model

Execute the following either through your CLI, within dbt Cloud, or within [Snowplow Console](/docs/modeling-your-data/running-data-models-via-console/)

```yml
dbt run --select snowplow_fractribution
```

### 5. Run the python script to generate the final models

Depending on your setup, please follow the appropriate steps below. All these methods will create the following tables in your warehouse:

- `snowplow_fractribution_path_summary_with_channels`
- `snowplow_fractribution_report_table`
- `snowplow_fractribution_channel_attribution`

**Run on Snowflake using Snowpark**

If you enabled this already, the tables will have already been built as part of step 4. If you wish to just re-run the attribution modeling for some reason you can run the following:

```text
dbt run --select snowplow_fractribution_call_snowpark_macros
```

**Locally run python**

> **Tip:** To run the fractribution script locally in Python, we recommend using a virtual environment such as one in `conda` or `pyenv`.
>
> Example using conda:
>
> ```text
> conda create --name fractribution_env -c https://repo.anaconda.com/pkgs/snowflake python=3.8 absl-py
>
> conda activate fractribution_env
> ```

#### I. Install packages

You can install the packages using `pip install -r dbt_packages/snowplow_fractribution/utils/requirements.txt` (or the appropriate path from your terminal working directory).

Please note that some of the libraries are adapter specific. These are listed in the `requirements` file, and you can also find the necessary list for each adapter below:

**BigQuery:**

- `absl-py`==`1.2.0`
- `google-cloud-bigquery`==`3.5.0`

**Databricks:**

- `absl-py`==`1.2.0`,
- `databricks-sql-connector`==`2.1.0`
- `pandas`

**Snowflake:**

- `absl-py`==`1.2.0`,
- `snowflake-snowpark-python`==`0.11.0`

**Redshift:**

- `absl-py`==`1.2.0`,
- `redshift_connector`==`2.0.910`

***

**M1 Instructions (for Snowflake only)**

> **Note:** There is an issue with running Snowpark on M1 chips. A workaround recommended by Snowflake is to set up a virtual environment that uses x86 Python:
>
> ```text
> CONDA_SUBDIR=osx-64 conda create -n fractribution_env python=3.8 absl-py -c https://repo.anaconda.com/pkgs/snowflake
>
> conda activate fractribution_env
>
> conda config --env --set subdir osx-64
> ```

#### II. Set the connection parameters in your terminal

**BigQuery:**

```text
export project_id=project_id\

export bigquery_dataset=bigquery_dataset\

export google_application_credentials=google_application_credentials
```

**Databricks:**

```text
export databricks_schema=derived_schema_name\

export databricks_server_hostname=hostname\

export databricks_http_path=http_path\

export databricks_token=token
```

**Snowflake:**

```text
export snowflake_account=my_account\

export snowflake_user=sf_user\

export snowflake_password=password\

export snowflake_user_role=special_role\

export snowflake_warehouse=warehouse_name\

export snowflake_database=database_name\

export snowflake_schema=derived_schema_name
```

**Redshift:**

```text
export redshift_host=redshift_host\

export redshift_database=redshift_database\

export redshift_port=redshift_port\

export redshift_user=redshift_user\

export redshift_password=redshift_password\

export redshift_schema=redshift_schema
```

***

#### III. Run the fractribution script

Run the adapter specific main fractribution script by specifying the conversion window start and end dates, and the attribution model (if you are not using the default `shapley`, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/#attribution-models) for more options). Example:

**BigQuery:**

```text
python main_snowplow_bigquery.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

**Databricks:**

```text
python main_snowplow_databricks.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

**Snowflake:**

```text
python main_snowplow_snowflake.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

**Redshift:**

```text
python main_snowplow_redshift.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

***

> **Tip:** To help debug easier we have included logs to be printed each time a table is created in the warehouse, there should be 3 in total (`snowplow_fractribution_path_summary_with_channels`, `snowplow_fractribution_channel_attribution` and `snowplow_fractribution_report_table`). There is also a `--verbose` flag you can include in your shell prompt when executing the python scripts which may give you an even more detailed breakdown depending on your warehouse.

**Run python with docker**

#### I. Pull the docker image ​

You can pull the latest docker image from Docker Hub: `docker pull snowplow/fractribution:latest`. Alternatively, you can pull it based on the package version: `docker pull snowplow/fractribution:0.2.0`

#### II. Set the environment variables

Add the necessary environment variables to an environment file, e.g. `configs.env`. The necessary variables will differ depending on the data warehouse you are using. The easiest way to determine the variables you need to set is to check the Dockerfile in the fractribution dbt package: `dbt-snowplow-fractribution/utils/Dockerfile`. Please note that in case of BigQuery, the `google_application_credentials` env var is not needed for Docker as you mount this as a volume at run time.

Below is an example of the `config.env` file (set up for Snowflake). Note the last 4 variables are named the same across all warehouses. You do not need to specify the attribution model if using the default, `shapley`:

```text
snowflake_account=youraccount.ap-southeast-2

snowflake_user=user

snowflake_password=abc123

snowflake_user_role=DBT

snowflake_warehouse=WH

snowflake_database=snowplow

snowflake_schema=FRACTRIBUTION_DERIVED



conversion_window_start_date=2022-06-03

conversion_window_end_date=2022-08-01

attribution_model=last_touch

warehouse=snowflake
```

#### III. Run the docker container

Run the docker container :

**BigQuery:**

With BigQuery you need to mount your service account keyfile when running the docker image

```text
docker run --rm --env-file /path/to/env/file/configs.env -v /path/to/yourkeyfile.json:/keyfile.json -it snowplow/fractribution:latest
```

**Databricks:**

```text
docker run --rm --env-file /path/to/env/file/configs.env -it snowplow/fractribution:latest
```

**Snowflake:**

```text
docker run --rm --env-file /path/to/env/file/configs.env -it snowplow/fractribution:latest
```

**Redshift:**

```text
docker run --rm --env-file /path/to/env/file/configs.env -it snowplow/fractribution:latest
```

***

> **Tip:** To help debug easier we have included logs to be printed each time a table is created in the warehouse, there should be 3 in total (`snowplow_fractribution_path_summary_with_channels`, `snowplow_fractribution_channel_attribution` and `snowplow_fractribution_report_table`). In order to run the script with a `--verbose` flag you can set the `verbose_run` variable to "true" in your env-file, which may give you an even more detailed breakdown depending on your warehouse.

---

# Quick start guides for the legacy dbt packages
> Quick start guides for legacy Snowplow dbt packages including Web, Mobile, and Fractribution packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/

> **Note:** Legacy packages are no longer under active development and are in some cases no longer supported. Please look for a suitable replacement for these packages where possible.

| Package name  | Purpose                                             | Superseded by                                                                                              |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Web           | Transform raw web event data into derived tables    | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Mobile        | Transform raw mobile event data into derived tables | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Fractribution | Marketing attribution analysis                      | [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) |

---

# Mobile Quickstart
> Quick start guide for the legacy Snowplow Mobile dbt package to model mobile event data into sessions and users.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/mobile/

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a mobile events dataset being available in your database, the requirements are:

- Snowplow [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) or [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) mobile tracker version 1.1.0 or later implemented.
- Mobile session context enabled ([ios](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/ios-tracker-1-7-0/#session-context) or [android](/docs/sources/mobile-trackers/previous-versions/android-tracker/android-1-7-0/#session-tracking)).
- Screen view events enabled ([ios](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/ios-tracker-1-7-0/#tracking-features) or [android](/docs/sources/mobile-trackers/previous-versions/android-tracker/android-1-7-0/#tracking-features)).

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_mobile

  version: 1.0.0
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the mobile model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-mobile/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile), in the table labeled `events`. In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_mobile:

    snowplow__atomic_schema: schema_with_snowplow_events

    snowplow__database: database_with_snowplow_events

    snowplow__events_table: table_of_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Enabled desired contexts

The mobile package has the option to join in data from the following 4 Snowplow contexts:

- Mobile context -- Device type, OS, etc.
- Geolocation context -- Device latitude, longitude, bearing, etc.
- Application context -- App version and build
- Screen context -- Screen details associated with mobile event

By default these are **all disabled** in the mobile package. Assuming you have the enrichments turned on in your Snowplow pipeline, to enable the contexts within the package please modify the following in your `dbt_project.yml` file:

```yml
vars:

  snowplow_mobile:

    snowplow__enable_mobile_context: true

    snowplow__enable_geolocation_context: true

    snowplow__enable_application_context: true

    snowplow__enable_screen_context: true
```

### 5. Enable desired modules

The mobile package has the option to join in data from the following 1 Snowplow module:

- App Errors module -- Details relating to app errors that occur during sessions

By default this module is **disabled** in the mobile package. Assuming you have the enrichments turned on in your Snowplow pipeline, to enable the module within the package please modify the following in your `dbt_project.yml` file:

```yml
vars:

  snowplow_mobile:

    snowplow__enable_app_errors_module: true
```

### 6. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add/modify the following in your `dbt_project.yml` file:

```yml
vars:

  snowplow_mobile:

    snowplow__start_date: 'yyyy-mm-dd'

    snowplow__app_id: ['my_app_1','my_app_2']
```

### 7. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>
>   snowplow_mobile:
>
>     snowplow__derived_tstamp_partitioned: false
> ```

> **Info:** Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>
>   snowplow_mobile:
>
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> **A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).**

### 8. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_mobile
```

---

# Web Quickstart
> Quick start guide for the legacy Snowplow Web dbt package to model web event data into page views, sessions, and users.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/web/

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web events dataset being available in your database, the requirements are:

- [Snowplow Javascript tracker](/docs/sources/web-trackers/) version 2 or later implemented.
- Web Page context [enabled](/docs/sources/web-trackers/tracker-setup/initialization-options/) (enabled by default in [v3+](/docs/sources/web-trackers/tracker-setup/initialization-options/)).
- [Page view events](/docs/sources/web-trackers/tracking-events/page-views/) implemented.
- From version v0.13.0 onwards you must be using [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v4.0.0 and above, or [BigQuery Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v1.0.0 and above. If you are not using these versions, or are using the Postgres loader, you will need to set `snowplow__enable_load_tstamp` to `false` in your `dbt_project.yml` and will not be able to use the consent models.

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_web

  version: 1.0.1
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the web model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-web/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_web:

    snowplow__atomic_schema: schema_with_snowplow_events

    snowplow__database: database_with_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Enabled desired contexts

The web package has the option to join in data from the following 3 Snowplow enrichments:

- [IAB enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/)
- [UA Parser enrichment](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)
- [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)

By default these are **all disabled** in the web package. Assuming you have the enrichments turned on in your Snowplow pipeline, to enable the contexts within the package please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_web:

    snowplow__enable_iab: true

    snowplow__enable_ua: true

    snowplow__enable_yauaa: true
```

### 5. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_web:

    snowplow__start_date: 'yyyy-mm-dd'

    snowplow__app_id: ['my_app_1','my_app_2']
```

### 6. Verify page ping variables

The web package processes page ping events to calculate web page engagement times. If your [tracker configuration](/docs/sources/web-trackers/tracking-events/activity-page-pings/) for `min_visit_length` (default 5) and `heartbeat` (default 10) differs from the defaults provided in this package, you can override by adding to your `dbt_project.yml`:

```yml
vars:

  snowplow_web:

    snowplow__min_visit_length: 5 # Default value

    snowplow__heartbeat: 10 # Default value
```

### 7. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>
>   snowplow_web:
>
>     snowplow__derived_tstamp_partitioned: false
> ```

> **Info:** Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>
>   snowplow_web:
>
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> **A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).**

### 8. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package). As this package contains some seed files, you will need to seed these first

```bash
dbt seed --select snowplow_web --full-refresh

dbt run --selector snowplow_web
```

### 9. Enable extras

The package comes with additional modules and functionality that you can enable, for more information see the [consent tracking](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/consent-module/), [conversions](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/conversions/), and [core web vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/core-web-vitals-module/) documentation.

---

# Media Player Quickstart
> Quick start guide for the Snowplow Media Player dbt package to model video and audio engagement data.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/media-player/

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web or mobile events dataset being available in your database:

- A dataset of media events must be available in the database. You can collect media events using our plugins for the JavaScript tracker or using the iOS and Android trackers: [Media plugin](/docs/sources/web-trackers/tracking-events/media/), [HTML5 media player plugin](/docs/sources/web-trackers/tracking-events/media/html5/), [YouTube plugin](/docs/sources/web-trackers/tracking-events/media/youtube/), [Vimeo plugin](/docs/sources/web-trackers/tracking-events/media/vimeo/) or the [iOS and Android media APIs](/docs/sources/mobile-trackers/tracking-events/media-tracking/)
- Have the [`webPage` context](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) enabled on Web or the [screen context](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-view-event-and-screen-context-entity) on mobile (enabled by default).
- Enabled session tracking on the tracker (default on mobile).

The model is compatible with all versions of our media tracking APIs. These have evolved over time and may track the media events using two sets of event and contexts schemas:

1. Version 1 media schemas:

   - [media-player event schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player_event/jsonschema/1-0-0) used for all media events.

   - [media-player context v1 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0).

   - Depending on the plugin / intention there are player-specific contexts:

     - in case of embedded YouTube tracking: Have the [YouTube specific context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.youtube/youtube/jsonschema/1-0-0) enabled.
     - in case of HTML5 audio or video tracking: Have the [HTML5 media element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/media_element/jsonschema/1-0-0) enabled.
     - in case of HTML5 video tracking: Have the [HTML5 video element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/video_element/jsonschema/1-0-0) enabled.

2. Version 2 media schemas (preferred):

   - [per-event media event schemas](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.media).
   - [media-player context v2 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0).
   - optional [media-session context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0).
   - optional [media-ad](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0) and [ad break](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0) context schema.

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_media_player

  version: 1.0.1
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the media player model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-media-player/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile), in the table labeled `events`. In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_media_player:

    snowplow__atomic_schema: schema_with_snowplow_events

    snowplow__database: database_with_snowplow_events

    snowplow__events_table: table_of_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Filter your data set

You can specify both `start_date` at which to start processing events, the `app_id`'s to filter for, and the `event_name` value to filter on. By default the `start_date` is set to `2020-01-01`, all `app_id`'s are selected, and all events with the `com.snowplowanalytics.snowplow.media` or the `media_player_event` event name are being surfaced. To change this please add/modify the following in your `dbt_project.yml` file:

```yml
...

vars:

  snowplow_media_player:

    snowplow__start_date: 'yyyy-mm-dd'

    snowplow__app_id: ['my_app_1','my_app_2']

    snowplow__media_event_names: ['media_player_event', 'my_custom_media_event']
```

### 5. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `load_tstamp` or `collector_tstamp`. Unless it is partitioned on `derived_tstamp` (legacy behavior) you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `snowplow__session_timestamp` column is used for partition pruning when querying the events table:
>
> ```yml
> ...
>
> vars:
>
>   snowplow_media_player:
>
>     snowplow__derived_tstamp_partitioned: false
> ```

### 6. Enable desired contexts and configuration

The media player package creates tables that depend on the existence of certain context entities that are tracked by the media plugins in the Snowplow trackers. Depending on which media plugin or tracking implementation you, you will need to enable the relevant contexts in your `dbt_project.yml`.

#### 6a. Using trackers with support for the version 2 media schemas

This option applied in case you are tracking media events with either the Snowplow Media plugin, Vimeo plugin for JavaScript tracker, or the iOS/Android trackers.

```yaml
...

vars:

  snowplow_media_player:

    # don't use the older version 1 of the media player context schema

    snowplow__enable_media_player_v1: false

    # use the version 2 of the media player context schema

    snowplow__enable_media_player_v2: true

    # use the media session context schema (unless disabled on the tracker)

    snowplow__enable_media_session: true

    # depending on whether you track ads, ad breaks and progress within ads:

    snowplow__enable_media_ad: true

    snowplow__enable_media_ad_break: true

    snowplow__enable_ad_quartile_event: true

    # depending on whether you track events from web or mobile apps:

    snowplow__enable_web_events: true

    snowplow__enable_mobile_events: true
```

#### 6b. Using the HTML5 media tracking plugin for JavaScript tracker

```yaml
...

vars:

  snowplow_media_player:

    # use the version 1 of the media player context schema used by the YouTube plugin

    snowplow__enable_media_player_v1: true

    # don't use the version 2 of the media player context schema as it is not tracked by the plugin

    snowplow__enable_media_player_v2: false

    # don't use the media session context schema as it is not tracked by the plugin

    snowplow__enable_media_session: false

    # set to true if the HTML5 media element context schema is enabled

    snowplow__enable_whatwg_media: true

    # set to true if the HTML5 video element context schema is enabled

    snowplow__enable_whatwg_video: true
```

#### 6c. Using the YouTube tracking plugin for JavaScript tracker

```yaml
...

vars:

  snowplow_media_player:

    # use the version 1 of the media player context schema used by the YouTube plugin

    snowplow__enable_media_player_v1: true

    # don't use the version 2 of the media player context schema as it is not tracked by the plugin

    snowplow__enable_media_player_v2: false

    # don't use the media session context schema as it is not tracked by the plugin

    snowplow__enable_media_session: false

    # set to true if the YouTube context schema is enabled

    snowplow__enable_youtube: true
```

For other variables you can configure please see the [variables](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#variables) section.

### 7. Optimize your project

There are ways how you can deal with [high volume optimizations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/) at a later stage, if needed, but you can do a lot upfront by selecting carefully which variable to use for `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. This timestamp column should ideally be set to the column your event table is partitioned on. It is defaulted to `collector_tstamp` but depending on your loader it can be the `load_tstamp` as the sensible value to use:

```yml
vars:

  snowplow_media_player:

    snowplow__session_timestamp: 'load_tstamp'
```

### 8. Verify your variables using our Config guides (Optional)

If you are unsure whether the default values set are good enough in your case or you would already like to maximize the potential of your models, you can dive deeper into the meaning behind our variables on our [Config](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/media-player/) page. It includes a [Config Generator](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/media-player/#config-generator) to help you create all your variable configurations, if necessary.

### 9. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_media_player
```

---

# Normalize Quickstart
> Quick start guide for the Snowplow Normalize dbt package to flatten and normalize events and entities for downstream tools.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/normalize/

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed:

- Python 3.7 or later

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_normalize

  version: 1.0.0
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the normalize model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-normalize/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_normalize:

    snowplow__atomic_schema: schema_with_snowplow_events

    snowplow__database: database_with_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add the following to your `dbt_project.yml` file:

```yml
vars:

  snowplow_normalize:

    snowplow__start_date: 'yyyy-mm-dd'

    snowplow__app_id: ['my_app_1','my_app_2']
```

> **Note:** If you have events you are going to normalize with no value for the `dvce_sent_tstamp` field, you need to disable the days late filter by setting the `snowplow__days_late_allowed` variable to `-1`, otherwise these events will not be processed.

### 5. Install additional python packages

The script only requires 2 additional packages (`jsonschema` and `requests`) that are not built into python by default, you can install these by running the below command, or by installing them by your preferred method.

```bash
pip install -r dbt_packages/snowplow_normalize/utils/requirements.txt
```

### 6. Setup the generator configuration file

You can use the example provided in `utils/example_normalize_config.json` to start your configuration file to specify which events, self-describing events, and contexts you wish to include in each table. For more information on this file see the [normalize package docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/).

### 7. Setup your resolver connection file _(optional)_

If you are not using iglu central as your only iglu registry then you will need to set up an [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) file and point to this in your generator config.

### 8. Generate your models

At the root of your dbt project, running `python dbt_packages/snowplow_normalize/utils/snowplow_normalize_model_gen.py path/to/your/config.json` will generate all models specified in your configuration.

### 9. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `load_tstamp` or `collector_tstamp`. Unless it is partitioned on `derived_tstamp` (legacy behavior) you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `snowplow__session_timestamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>
>   snowplow_normalize:
>
>     snowplow__derived_tstamp_partitioned: false
> ```

> **Info:** Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>
>   snowplow_normalize:
>
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> **A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).**

### 10. Change the default partition timestamp _(optional)_

The package uses a configurable partition timestamp column, controlled by the `snowplow__partition_tstamp` variable:

```yaml
vars:

  snowplow__partition_tstamp: "collector_tstamp"  # Default value, any change should be a timestamp
```

The purpose of this variable is to adjust the partitioning of the derived tables to use a different timestamp (e.g., derived\_tstamp) that is more suitable for analytics in the next layer.

> **Warning:** If you change `snowplow__partition_tstamp` to a different column (e.g., `load_tstamp`), you MUST ensure that this column is included in the `event_columns` list in your normalize configuration for each event. Failing to do so will cause the models to fail, as the partition column must be present in the normalized output.
>
> Example configuration when using a custom partition timestamp:
>
> ```json
> {
>
>     "events": [
>
>         {
>
>             "event_names": ["page_view"],
>
>             "event_columns": [
>
>                 "domain_userid",
>
>                 "load_tstamp",  // Must include your custom partition timestamp here
>
>                 "app_id"
>
>             ],
>
>             // ... rest of configuration
>
>         }
>
>     ]
>
> }
> ```

### 11. Run your model(s)

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_normalize
```

---

# Unified Digital Quickstart
> Quick start guide for the Snowplow Unified Digital dbt package to model web and mobile event data into views, sessions, and users.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/unified/

> **Info:** We are gradually moving tutorials like the QuickStart Guides from our standard Documentation to the **Tutorials & Guides** section. You can now find the latest tutorial on how to get started with the Unified Digital package [here](/tutorials/unified-digital/intro/).

---

# Utils Quickstart
> Quick start guide for the Snowplow Utils dbt package with core macros and helper functions for Snowplow dbt packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/utils/

> **Info:** The models, functionality, and variables described below are only available from `snowplow-utils v0.15.0` and above, as earlier packages do not utilize these variables.

The `snowplow-utils` package allows you to create your own custom `snowplow_base_events_this_run` table using macros that generate the required SQL code for you, allowing you to incorporate whatever custom event types, contexts of Snowplow data. Using this package will allow you to leverage the [incremental nature](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of the Snowplow packages, meaning you can more easily build data models using our other packages such as `snowplow-web` and `snowplow-mobile`, as well as build your own completely custom packages without having to do the initial heavy lifting yourself. This can, however, be a bit complicated to set up and so for that purpose we've create this quickstart page to guide you through this process.

> **Info:** It is only recommended that you use this if you are planning on heavily customizing your Snowplow data modeling setup, whilst still taking advantage of the incremental framework that the existing dbt packages offer. If you are going to be heavily leveraging the existing Snowplow packages (e.g. snowplow-unified) then you will not need to leverage this package for the creation of your base tables. Please instead follow the appropriate quickstart guides for the packages you are going to be utilizing instead, such as for [web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/web/) or [mobile](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/mobile/).

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and connected to your database you need:

- A dataset of Snowplow events from one of the [Snowplow trackers](/docs/sources/)

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml


packages:

- package: snowplow/snowplow_utils

  version: 1.0.0
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

> **Info:** You can largely skip redundant copy + pasting by cloning the following dbt project repository that we have created in GitHub. You can find it [here](https://github.com/snowplow-incubator/dbt-example-project), and this has all of the boilerplate setup for you already. If you want to customize model names or parameter values, you can still follow the quickstart guide below to help you understand how to do that, and what changing each variable will mean for your models. Feel free to skip steps 1 and 2, however.

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Creating the `base` module in your dbt project

In your dbt project, create a `base` folder somewhere within your `models` directory. Within this `base` folder, you need to create the 6 `.sql` files shown below:

```ascii
my_dbt_project

├── analyses

│   └── .gitkeep

├── dbt_packages

│   ├── dbt_utils

│   └── snowplow_utils

├── logs

├── macros

├── models

│   └── base

│       ├── snowplow_base_quarantined_sessions.sql

│       ├── snowplow_incremental_manifest.sql

│       ├── snowplow_base_new_event_limits.sql

│       ├── snowplow_base_sessions_lifecycle_manifest.sql

│       ├── snowplow_base_sessions_this_run.sql

│       └── snowplow_base_events_this_run.sql

├── seeds

├── snapshots

├── target

├── tests

├── .gitignore

├── dbt_project.yml

├── packages.yml

├── README.md

└── selectors.yml
```

Once you've created all of these models, you need to call the correct macros in each model to ensure that the correct SQL gets generated for each model. If you'd like to rename any of the models, all you need to do is rename the `.sql` files listed above.

> **Tip:** Please only rename the models with caution and be sure to read the subsequent steps carefully as you may need to modify some of the boilerplate code outlined below to have the macros adapt properly to your naming conventions.

### 3. Setting up the quarantined sessions macro

Within the `snowplow_base_quarantined_sessions.sql` file, you can call the `base_create_snowplow_quarantined_sessions` macro as follows:

```jinja2
{{

  config(

    materialized='incremental'

  )

}}



{% set quarantined_query = snowplow_utils.base_create_snowplow_quarantined_sessions() %}



{{ quarantined_query }}
```

This macro doesn't accept any arguments, and simply generates a table which contains a column named `session_identifier`, containing all session identifiers of sessions that have been quarantined due to exceeding the maximum session length, to avoid long table scans. More information on the sessionization logic and optimization can be found [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/)

### 4. Setting up the incremental manifest macro

Next, within the `snowplow_incremental_manifest.sql` file, you can call the `base_create_snowplow_incremental_manifest` macro as follows:

```jinja2
{{

  config(

    materialized='incremental'

  )

}}



{% set incremental_manifest_query = snowplow_utils.base_create_snowplow_incremental_manifest() %}



{{ incremental_manifest_query }}
```

Much like with the quarantined sessions macro, this does not require any arguments and generates the boilerplate for the incremental manifest table that Snowplow leverages.

### 5. Setting up the new event limits macro

For the `snowplow_base_new_event_limits` model, you need to add a few extra macros into the mix, which you can do as follows:

> **Info:** Be sure to specify your `PACKAGE_NAME` when calling the `get_enabled_snowplow_models` macro.

```jinja2
{{

  config(

   post_hook=["\\{\\{snowplow_utils.print_run_limits(this)}}\\}"]

   )

}}



{%- set models_in_run = snowplow_utils.get_enabled_snowplow_models(PACKAGE_NAME, graph_object=none, models_to_run="", base_events_table_name='snowplow_base_events_this_run') -%}



{% set min_last_success,

         max_last_success,

         models_matched_from_manifest,

         has_matched_all_models = snowplow_utils.get_incremental_manifest_status(ref('snowplow_incremental_manifest'),

                                                                                 models_in_run) -%}





{% set run_limits_query = snowplow_utils.get_run_limits(min_last_success,

                                                          max_last_success,

                                                          models_matched_from_manifest,

                                                          has_matched_all_models,

                                                          var("snowplow__start_date")) -%}





{{ run_limits_query }}
```

Here there are a couple of variables that can be used to modify the `new_event_limits` table setup. Firstly, if you chose to name your `snowplow_incremental_manifest` model differently, be sure to reference that properly in the `get_incremental_manifest_status` macro call. If for example, you chose to call the manifest model `incremental_manifest` by naming your file `incremental_manifest.sql` instead of the prescribed `snowplow_incremental_manifest.sql`, then you would reflect that using the following macro call:

> **Info:** The below is just an example of a macro call when using custom naming, this isn't something additional to copy and add if you want to set up your `snowplow_new_event_limits` model!

```jinja2
{% set min_last_success,

         max_last_success,

         models_matched_from_manifest,

         has_matched_all_models = snowplow_utils.get_incremental_manifest_status(ref('incremental_manifest'),

                                                                                 models_in_run) -%}
```

Secondly, you can choose the starting date of when your Snowplow data was first loaded into your data warehouse/lake. This is reflected in the value of the `snowplow__start_date` variable, which you may be familiar with if you've used previous Snowplow dbt packages. This variable is typically defined in your `dbt_project.yml`, and can be defined in that file in the following manner.

```yml


vars:

    # to define it globally, use the following notation

    snowplow__start_date: '2020-01-01'



    # to define the variable locally within the project, use the following nesting

    my_dbt_project:

        snowplow__start_date: '2020-01-01'
```

### 6. Setting up the sessions lifecycle manifest macro

For the `snowplow_base_sessions_lifecycle_manifest` model, you have the following macro call which takes in a lot of parameters to allow for a high level of flexibility in how you can process your Snowplow data:

```jinja2


{% set sessions_lifecycle_manifest_query = snowplow_utils.base_create_snowplow_sessions_lifecycle_manifest(

    session_identifiers=var('snowplow__session_identifiers', '[\\{"schema": "atomic", "field": "domain_sessionid"}]'),

    session_sql=var('snowplow__session_sql', none),

    session_timestamp=var('snowplow__session_timestamp', 'collector_tstamp'),

    user_identifiers=var('snowplow__user_identifiers', '[\\{"schema": "atomic", "field": "domain_userid"}]'),

    user_sql=var('snowplow__user_sql', none),

    quarantined_sessions=var('snowplow__quarantined_sessions', 'snowplow_base_quarantined_sessions'),

    derived_tstamp_partitioned=var('snowplow__derived_tstamp_partitioned', true),

    days_late_allowed=var('snowplow__days_late_allowed', 3),

    max_session_days=var('snowplow__max_session_days', 3),

    app_ids=var('snowplow__app_ids', []),

    snowplow_events_database=var('snowplow__events_schema', none),

    snowplow_events_schema=var('snowplow__events_schema', 'atomic'),

    snowplow_events_table=var('snowplow__events_table', 'events'),

    event_limits_table=var('snowplow__event_limits', 'snowplow_base_new_event_limits'),

    incremental_manifest_table=var('snowplow__incremental_manifest', 'snowplow_incremental_manifest'),

    package_name=var('snowplow__package_name, 'snowplow')

 ) %}



{{ sessions_lifecycle_manifest_query }}
```

To get an in-depth explanation of each variable passed here, please refer to the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/) page.

There are several important parameters to consider. The first one is `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. It's recommended to use either `collector_tstamp` or `load_tstamp` as sensible values.

Next, we have `snowplow__session_identifiers` and `snowplow__user_identifiers`, which expect a map object defining the location of session or user identifiers. In this map, the key represents the name of the context or entity where the identifier can be found. If the identifier is a field in the atomic columns, the key can be set to `atomic`. The value specifies the name of the field in either the context/entity or the atomic columns.

> **Note:** Currently, we only support session and user identifiers found in atomic fields for Redshift/Postgres. We don't support nested level fields for any warehouses, and for BigQuery you will currently need to do the version management yourself. We will be getting around to supporting this extra functionality soon.

By default, `snowplow__session_identifiers` is set to `[\\{"schema": "atomic", "field": "domain_sessionid"}]`, and `snowplow__user_identifiers` is set to `[\\{"schema": "atomic", "field": "domain_userid"}]`. This means that the identifiers for sessions and users are expected to be found in the `domain_sessionid` and `domain_userid` fields, respectively.

If you have more than one session or user identifier, you can specify multiple entries in the map. The order in which you list them determines the precedence that the macro will use to look for these field values, and `COALESCE` them into the common session/user\_identifier field. E.g. if you have the following definition for your `user_identifier`:

**BigQuery, Databricks, & Snowflake:**

```json

[
\{"schema": "my_custom_context", "field": "internal_user_id"},
\{"schema": "atomic", "field": "domain_userid"}
]
```

**Redshift & Postgres:**

```json

[
\{"schema": "my_custom_context", "field": "internal_user_id", "prefix": "mcc", "alias": "mcc_iud"},
\{"schema": "atomic", "field": "domain_userid"}
]
```

For Redshift & Postgres we also introduce the `prefix` and `alias` fields, where `prefix` is the `prefix` that is put in front of each field name in the context, and `alias` is the table alias used upon joining. This can be useful when you are using custom SQL. As an example, using the above configuration we could access the `internal_user_id` field using the following SQL:

```sql
mcc_iud.mcc_internal_user_id as internal_user_id,
```

This could be leveraged in the `snowplow__custom_sql` variable. For more examples, please see [Custom Sessionization & Users](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/).

***

The package will first extract `internal_user_id` from the `my_custom_context` context, and then use something similar to the following SQL statement: `COALESCE(my_custom_context.internal_user_id, events.domain_userid) as user_identifier`. This way, if a user is able to identify themselves through logging in which would populate a context called `my_custom_context`, their `internal_user_id` is used as a `user_identifier`. If, however, this is not the case, then the `user_identifier` field falls back on the value that the `domain_userid` has.

> **Info:** We currently only track one `user_identifier` value per session in the `session_lifecycle_manifest`, which means if a user logs in part-way through a session, we would only keep one of those values.

Further, you can specify some additional configurations here such as in which table/schema the events data sits, what the names are of your `event_limits` and `incremental_manifest` tables, and some parameters around what the maximum session length is. You should once again be familiar with the majority of these variables if you've used another of our dbt packages before.

### 7. Setting up the sessions this run macro

For the `snowplow_base_sessions_this_run` model, you will need to add a post-hook to the configuration of the model as follows:

```jinja2
{{

    config(

        post_hook=["\\{\\{ snowplow_utils.base_quarantine_sessions(var('snowplow__max_session_days', 3), var('snowplow__quarantined_sessions', 'snowplow_base_quarantined_sessions')) }}\\}"]

    )

}}





{% set sessions_query = snowplow_utils.base_create_snowplow_sessions_this_run(

    lifecycle_manifest_table='snowplow_base_sessions_lifecycle_manifest',

    new_event_limits_table='snowplow_base_new_event_limits') %}



{{ sessions_query }}
```

Here the parameters that are called in both macros are only used to direct the macro to the right model names, so again if you've chosen to modify any of the table names then you should adjust the names in the right macros here. For the `base_quarantine_sessions` macro you simply pass the maximum session duration in days, which is taken from the `snowplow__max_session_days` variable, and you specify the name of the `snowplow_base_quarantined_sessions` table, specified by the `snowplow__quarantined_sessions` variable.

For the `base_create_snowplow_sessions_this_run` macro call, you specify the name of the `lifecycle_manifest_table` and the `new_event_limits_table`. The boilerplate contains their default names, and so if you have not customized anything you can simply copy this code into your `snowplow_base_sessions_this_run` model.

### 8. Setting up the events this run macro

For the `snowplow_base_events_this_run` model, you will need to run the following two macros in your model:

```jinja2
{%- set lower_limit, upper_limit = snowplow_utils.return_limits_from_model(ref(var('snowplow__base_sessions', 'snowplow_base_sessions_this_run')),

                                                                          'start_tstamp',

                                                                          'end_tstamp') %}



{% set base_events_query = snowplow_utils.base_create_snowplow_events_this_run(

    sessions_this_run_table=var('snowplow__base_sessions', 'snowplow_base_sessions_this_run'),

    session_identifiers=var('snowplow__session_identifiers', '[\\{"schema": "atomic", "field": "domain_sessionid"}]'),

    session_sql=var('snowplow__session_sql', none),

    session_timestamp=var('snowplow__session_timestamp', 'collector_tstamp'),

    derived_tstamp_partitioned=var('snowplow__derived_tstamp_partitioned', true),

    days_late_allowed=var('snowplow__days_late_allowed', 3),

    max_session_days=var('snowplow__max_session_days', 3),

    app_ids=var('snowplow__app_ids', []),

    snowplow_events_database=var('snowplow__events_schema', none),

    snowplow_events_schema=var('snowplow__events_schema', 'atomic'),

    snowplow_events_table=var('snowplow__events_table', 'events')) %}



{{ base_events_query }}
```

Here you once again have a number of parameters that the macro can take, and to get an in-depth explanation of each variable passed here, please refer to the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/) page. The variables used here are largely either self-explanatory or overlapping with those in the [lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/utils/#6-setting-up-the-sessions-lifecycle-manifest-macro) section, except that you can now specify custom names for your `snowplow_base_sessions_this_run` table through the `snowplow__base_sessions` variable.

### 9. Modify your `dbt_project.yml`

To properly configure your dbt project to utilize and update the manifest tables correctly, you will need to add the following hooks to your `dbt_project.yml`

```yml
# Completely or partially remove models from the manifest during run start.

on-run-start:

  - "\\{\\{ snowplow_utils.snowplow_delete_from_manifest(var('models_to_remove',[]), ref('snowplow_incremental_manifest')) }}\\}"



# Update manifest table with last event consumed per sucessfully executed node/model

on-run-end:

  - "\\{\\{ snowplow_utils.snowplow_incremental_post_hook(package_name='snowplow', incremental_manifest_table_name=var('snowplow__incremental_manifest', 'snowplow_incremental_manifest'), base_events_this_run_table_name='snowplow_base_events_this_run', session_timestamp=var('snowplow__session_timestamp')) }}\\}"
```

The `snowplow_delete_from_manifest` macro is called to remove models from manifest if specified using the `models_to_remove` variable, in case of a partial or full refresh. The `snowplow_incremental_post_hook` is used to update the manifest table with the timestamp of the last event consumed successfully for each Snowplow incremental model - make sure to change the `base_events_this_run_table_name` if you used a different table name.

> **Tip:** The `package_name` variable here is not necessarily the name of your project (although it keeps things simple to make it the same), instead it is what is used to identify your tagged incremental models as they should be tagged with `<package_name>_incremental`.

### 10. Run your models

Now that you've configured all of your macros correctly, you can run your dbt project as normal. If you want to try to only run the models you've just created above, you can issue the following command in your terminal:

```bash
dbt run --select +snowplow_base_events_this_run
```

You will have to change the name of the model if you've customized this earlier.

---

# Model your data with dbt
> Information for our dbt packages including quickstarts, configurations, and building custom models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/

[dbt](https://docs.getdbt.com/) enables analytics engineers to transform data in their warehouses by simply writing select statements. Snowplow has written and maintain a number of dbt packages to model your snowplow data for various purposes and produce derived tables for use in analytics, AI, ML, BI, or reverse ETL tools.

Using Snowplow's dbt packages means you can draw insight and value from your data _quicker_, _easier_, and _cheaper_ than building your own modeling from scratch.

![Snowplow Data Modeling Packages](/assets/images/dbt_packages-light.drawio-acc485891071f9c2f66eb4039698e29f.png)

To set up dbt, Snowplow Self-Hosted users can start with the [dbt User Guide](https://docs.getdbt.com/guides/getting-started) and then we have prepared some [introduction videos](https://www.youtube.com/watch?v=1kd6BJhC4BE) for working with the Snowplow dbt packages.

For Snowplow CDI customers, dbt projects can be configured and scheduled in the console meaning you can [get started](/docs/modeling-your-data/running-data-models-via-console/) running dbt models alongside your Snowplow pipelines.

## Snowplow dbt Packages

> **Info:** Our dbt packages are available under a mix of licenses. For more information about how to get access to these packages, please contact us by requesting a demo if you are new to Snowplow or by reaching out to our support team.

Our dbt packages come with powerful built-in features such as an [optimization to the incremental materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) to save you cost on warehouse compute resources compared to the standard method, a custom [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) to ensure we process just the required data for each run and keep your models in sync, plus the ability to build your own custom models using both of these!

There are 4 core snowplow dbt packages:

- [Snowplow Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-unified/#!/overview/snowplow_unified)): for modeling your web and mobile data for page and screen views, sessions, users, and consent
- [Snowplow Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-media-player/#!/overview/snowplow_media_player)): for modeling your media elements for play statistics
- [Snowplow Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-ecommerce/#!/overview/snowplow_ecommerce)): for modeling your Ecommerce interactions across carts, products, checkouts, and transactions
- [Snowplow Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-attribution/#!/overview/attribution)): used for Attribution Modeling with Snowplow

We also have 2 utility packages:

- [Snowplow Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/) package that makes it easy for you to build models that transform your events data into a different structure that may be better suited for downstream consumers
- [Snowplow Utils](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/) contains all our shared macros and features used across our packages

There are also 3 legacy dbt packages for web, mobile (superseded by unified) and fractribution (superseeded by attribution):

- [Snowplow Web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-web/#!/overview/snowplow_web)): for modeling your web data for page views, sessions, users, and consent
- [Snowplow Mobile](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-mobile/#!/overview/snowplow_mobile)): for modeling your mobile app data for screen views, sessions, users, and crashes
- [Snowplow Fractribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-fractribution/#!/overview/fractribution)): used for Attribution Modeling with Snowplow

Each package comes with a set of standard models to take your [Snowplow tracker data](/docs/sources/) and produce tables aggregated to different levels, or to perform analysis for you. You can also add your own models on top, see the page on [custom modules](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) for more information on how to do this.

## Package Versions & dbt Compatibility

Starting with v1.0.0, Snowplow dbt packages adopt stricter syntax requirements introduced in dbt Core ≥ 1.10.6 (for example, nested generic test arguments). Earlier dbt versions are not supported from this release onward. Additionally, for Redshift users only, redshift-adapters 1.10 is the minimum requirement going forward.

Snowplow maintains two active release lines:

- v1.x (latest) – recommended for all users, it is where ongoing development occurs
- v0.x (maintenance) – receives critical bug fixes only, where possible

For users upgrading from v0.x to v1.x, see our [migration guides](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/#upgrading-to-100)

### Note on dbt Fusion

While these packages are architecturally compatible with the dbt Fusion engine, Snowplow packages are not officially supported on dbt Fusion at this time. Limited testing may be possible using the [--no-version-check](https://docs.getdbt.com/reference/project-configs/require-dbt-version#fusion-compatibility) flag. Behavior can be fragile, particularly during full-refreshes or when introducing new models, and is under investigation.

The supported data warehouses per version can be seen below:

**Snowplow Unified Digital:**

| snowplow-unified version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ------------------------ | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                    | >=1.10.6 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.4.5                    | >=1.6.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |

**Snowplow Media Player:**

| snowplow-media-player version | snowplow-web version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ----------------------------- | -------------------- | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.1                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.8.0                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.5.3                         | >=0.14.0 to <0.16.0  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.2                         | >=0.13.0 to <0.14.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.1                         | >=0.12.0 to <0.13.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.3.4                         | >=0.9.0 to <0.12.0   | >=1.0.0 to <1.3.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.1.0                         | >=0.6.0 to <0.7.0    | >=0.20.0 to <1.1.0 | ❌        | ❌          | ✅        | ❌         | ✅        | ❌     |

**Snowplow Normalize:**

| snowplow-normalize version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ✅     |
| 0.3.5                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.3                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.1.0                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

**Snowplow Ecommerce:**

| snowplow-ecommerce version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.1                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ✅     |
| 0.8.2                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ❌     |
| 0.3.0                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.1                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

Postgres is technically supported in the models within the package, however one of the contexts’ names is too long to be loaded via the Postgres Loader.

**Snowplow Attribution:**

| snowplow-attribution version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ---------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ✅     |
| 0.3.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ❌     |

***

## dbt Version Compatibility Checker

You may be using other dbt packages that require you to use a specific version of dbt, or a specific version of the `dbt_utils` package. Or you simply may not wish to upgrade your existing setup.

Here’s a way to check the latest version of our packages you can install for a given setup. Simply enter your dbt version and (optionally) `dbt_utils` version below.

_Please enter a valid dbt version._

---

# Migration guide for Attribution
> Migration guide for upgrading the Snowplow Attribution dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/attribution/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- For a full upgrade walkthrough, please follow the [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.6.0

- We had to introduce limitations for Redshift users:

  1. users are only allowed to have 1 path transformation defined within the `snowplow__path_transforms` variable
  2. the `snowplow__path_transforms` variable parameters only accept single-item arrays for `remove_if_last_and_not_all` and `remove_if_not_all`

- This is due to Redshift deprecating Python UDFs; the transformations now rely on native SQL, which allows less flexibility for automation.

- Existing users can stay on older versions of the package as long as their UDFs remain in the database.

```yml
vars:

  snowplow_attribution:

    snowplow__path_transforms: {'exposure_path': null}
```

### Upgrading to 0.5.0

- From now on the `snowplow__path_transforms` variable parameters only accept non-empty arrays for `remove_if_last_and_not_all` and `remove_if_not_all` variables instead of strings, please your variable overwrites in your dbt\_project.yml accordingly. Previously you could only remove one specific channel or campaign, now you can do multiple, if needed.

```yml
vars:

  snowplow_attribution:

    snowplow__path_transforms: {'exposure_path': null, 'remove_if_last_and_not_all': ['channel_to_remove_1', 'campaign_to_remove_1, 'campaign_to_remove_2']}
```

---

# Migration guide for Ecommerce
> Migration guide for upgrading the Snowplow Ecommerce dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/ecommerce/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- For a full upgrade walkthrough, please follow the [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.6.0

Note the minimum `dbt-core` version now required is 1.5.0

Most upgrades and new features should be available automatically after the first run of the new version of the package, however there are a few breaking changes to manage before this first run. To enable any other new optional features or make use of the new configuration options, please see the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/) page.

Both the quarantined sessions and the sessions lifecycle table have has a column renamed, please run the below statements to rename this column in your warehouse if not doing a full-refresh of the package.

```sql
alter table <your_schema>_snowplow_manifest.snowplow_ecommerce_base_sessions_lifecycle_manifest rename column session_id to session_identifier;

alter table <your_schema>_snowplow_manifest.snowplow_ecommerce_base_quarantined_sessions rename column session_id to session_identifier;
```

### Upgrading to 0.5.0

**This version requires a full refresh run if you have been using any previous versions.** You will not be able to upgrade and have the package work without doing a full refresh.

To enable modeling mobile events, set `snowplow__enable_mobile_events` to `true`.

### Upgrading to 0.4.2

Two of the derived tables need to be altered for existing Snowflake, Databricks or Redshift users. Please modify the below script to fit your schemas and apply them before running the upgraded package.

The other option is to do a [complete refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.

**SQL scripts**

**snowflake:**

```sql
alter table (your_schema)_derived.snowplow_ecommerce_cart_interactions alter column cart_total_value type decimal(9,2);

alter table (your_schema)_derived.snowplow_ecommerce_transaction_interactions alter column transaction_revenue type decimal(9,2), transaction_discount_amount type decimal(9,2), transaction_shipping type decimal(9,2), transaction_tax type decimal(9,2);
```

**databricks:**

```sql
create table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new

(

    event_id               STRING,

    page_view_id           STRING,

    domain_sessionid       STRING,

    event_in_session_index INT,

    domain_userid          STRING,

    network_userid         STRING,

    user_id                STRING,

    ecommerce_user_id      STRING,

    derived_tstamp         TIMESTAMP,

    derived_tstamp_date    DATE,

    cart_id                STRING,

    cart_currency          STRING,

    cart_total_value       DECIMAL(9, 2),

    cart_created           BOOLEAN,

    cart_emptied           BOOLEAN,

    cart_transacted        BOOLEAN,

    ecommerce_action_type  STRING

);



insert into (your_schema)_derived.snowplow_ecommerce_cart_interactions_new  select * from (your_schema)_derived.snowplow_ecommerce_cart_interactions;

drop table (your_schema)_derived.snowplow_ecommerce_cart_interactions;

create table (your_schema)_derived.snowplow_ecommerce_cart_interactions select * from (your_schema)_derived.snowplow_ecommerce_cart_interactions_new;

drop table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new;



create table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new

(

    event_id                    STRING,

    page_view_id                STRING,

    domain_sessionid            STRING,

    event_in_session_index      INT,

    domain_userid               STRING,

    network_userid              STRING,

    user_id                     STRING,

    ecommerce_user_id           STRING,

    derived_tstamp              TIMESTAMP,

    derived_tstamp_date         DATE,

    transaction_id              STRING,

    transaction_currency        STRING,

    transaction_payment_method  STRING,

    transaction_revenue         DECIMAL(9, 2),

    transaction_total_quantity  INT,

    transaction_credit_order    BOOLEAN,

    transaction_discount_amount DECIMAL(9, 2),

    transaction_discount_code   STRING,

    transaction_shipping        DECIMAL(9, 2),

    transaction_tax             DECIMAL(9, 2),

    ecommerce_user_email        STRING,

    ecommerce_user_is_guest     BOOLEAN,

    number_products             BIGINT

);



insert into (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new    select * from (your_schema)_derived.snowplow_ecommerce_transaction_interactions;

drop table (your_schema)_derived.snowplow_ecommerce_transaction_interactions;

create table (your_schema)_derived.snowplow_ecommerce_transaction_interactions select * from (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new ;

drop table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new;
```

**redshift:**

```sql
create table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new

(

    event_id CHAR(36)   ENCODE lzo

    ,page_view_id VARCHAR(4096)   ENCODE lzo

    ,domain_sessionid CHAR(128)   ENCODE lzo

    ,event_in_session_index BIGINT   ENCODE az64

    ,domain_userid VARCHAR(128)   ENCODE lzo

    ,network_userid VARCHAR(128)   ENCODE lzo

    ,user_id VARCHAR(255)   ENCODE lzo

    ,ecommerce_user_id VARCHAR(128)   ENCODE lzo

    ,derived_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

    ,derived_tstamp_date DATE   ENCODE az64

    ,cart_id VARCHAR(4096)   ENCODE lzo

    ,cart_currency CHAR(3)   ENCODE lzo

    ,cart_total_value NUMERIC(9,2)   ENCODE az64

    ,cart_created BOOLEAN   ENCODE RAW

    ,cart_emptied BOOLEAN   ENCODE RAW

    ,cart_transacted BOOLEAN   ENCODE RAW

    ,ecommerce_action_type VARCHAR(16)   ENCODE lzo

)



insert into (your_schema)_derived.snowplow_ecommerce_cart_interactions_new  select * from (your_schema)_derived.snowplow_ecommerce_cart_interactions;

drop table (your_schema)_derived.snowplow_ecommerce_cart_interactions;

alter table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new

rename to snowplow_ecommerce_cart_interactions;



create table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new

(

    event_id CHAR(36)   ENCODE lzo

    ,page_view_id VARCHAR(4096)   ENCODE lzo

    ,domain_sessionid CHAR(128)   ENCODE lzo

    ,event_in_session_index BIGINT   ENCODE az64

    ,domain_userid VARCHAR(128)   ENCODE lzo

    ,network_userid VARCHAR(128)   ENCODE lzo

    ,user_id VARCHAR(255)   ENCODE lzo

    ,ecommerce_user_id VARCHAR(128)   ENCODE lzo

    ,derived_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

    ,derived_tstamp_date DATE   ENCODE az64

    ,transaction_id VARCHAR(4096)   ENCODE lzo

    ,transaction_currency CHAR(3)   ENCODE lzo

    ,transaction_payment_method VARCHAR(128)   ENCODE lzo

    ,transaction_revenue NUMERIC(9,2)   ENCODE az64

    ,transaction_total_quantity INTEGER   ENCODE az64

    ,transaction_credit_order BOOLEAN   ENCODE RAW

    ,transaction_discount_amount NUMERIC(9,2)   ENCODE az64

    ,transaction_discount_code VARCHAR(99)   ENCODE lzo

    ,transaction_shipping NUMERIC(9,2)   ENCODE az64

    ,transaction_tax NUMERIC(9,2)   ENCODE az64

    ,ecommerce_user_email VARCHAR(256)   ENCODE lzo

    ,ecommerce_user_is_guest BOOLEAN   ENCODE RAW

    ,number_products BIGINT   ENCODE az64

)



insert into (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new    select * from (your_schema)_derived.snowplow_ecommerce_transaction_interactions;

drop table (your_schema)_derived.snowplow_ecommerce_transaction_interactions;

alter table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new rename to snowplow_ecommerce_transaction_interactions;
```

***

### Upgrading to 0.4.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

### Upgrading to 0.3.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Fractribution
> Migration guide for upgrading the legacy Snowplow Fractribution dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/fractribution/

This is a legacy package.

### Upgrading to 0.3.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)
- Other changes required by [snowplow-web version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/#upgrading-to-0140), in particular the deduplication logic for Redshift and Postgres users

### Upgrading to 0.2.0

- Variable names are now prefixed with `snowplow__`, please align the new ones found in the `dbt_project.yml` file
- `snowplow__path_transforms` variable is a dictionary instead of an array and that the path transform names have also changed (e.g: `Exposure` -> `exposure_path`). See docs for the latest values
- Version 1.3.0 of `dbt-core` now required
- The python scripts have changed and been renamed per warehouse, please ensure you run the correct version and/or pull the latest docker image

---

# Migration guides for dbt data models
> Migration guides for upgrading from previous versions of Snowplow dbt packages including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/

With each package release, new features and fixes become available, as well as changes to existing functionalities. In many cases, including some breaking changes, nothing needs to be done by the user of the packages to use these new features beyond simply installing the new version, although the logic within the package may have changed. All such changes are always listed in the package changelog and release posts on [Community](https://community.snowplow.io/).

In some cases certain core dependencies (such as the version of dbt required) or names of variables may change that will require the user to make changes to their models and/or configuration of the package. These are listed in the release post and changelog, and repeated in the following pages.

---

# Migration guide for Media Player
> Migration guide for upgrading the Snowplow Media Player dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/media-player/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- For a full upgrade walkthrough, please follow the [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.8.0

**We recommend a [full refresh run](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) if you have been using any previous versions.** There might be inconsistencies in the calculation of impressions and play rate in the media stats table without doing a full refresh.

**Breaking changes:**

- The calculation of impressions in the stats table changed to use distinct plays instead of page views. This allows for multiple videos on the same page to be counted as separate impressions.

### Upgrading to 0.7.0

**This version requires a [full refresh run](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) if you have been using any previous versions.** You will not be able to upgrade and have the package work without doing a full refresh.

**Breaking changes:**

- A new and more robust `media_identifier` field to replace using `media_id` as a key in the derived tables.
- The introduction of new base macro functionality means in places the session and user identifier fields have been renamed to `session_identifier` and `user_identifier`.
- The default session identifier has been updated from using the `domain_sessionid`, to now be the media session id (or the page/screen view id if the media session entity is not set).
- Adds a primary key, `media_ad_view_id`, to the ad views table.

### Upgrading to 0.6.0

**This version requires a [full refresh run](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) if you have been using any previous versions.** You will not be able to upgrade and have the package work without doing a full refresh.

Please [check the quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/media-player/) for a guide on which new configuration options to enable. If you remain tracking events using the HTML5 and YouTube tracking plugins for the JavaScript tracker, you should add the following configuration to your `dbt_project.yml`:

```yml
vars:

  snowplow_media_player:

    # enable the version 1 player property

    snowplow__enable_media_player_v1: true

    # disable the new version 2 schemas

    snowplow__enable_media_player_v2: false

    snowplow__enable_media_session: false
```

On Redshift, the context names no longer require the use of the `{{ source('...') }}` macro, the table names can be passed directly.

### Upgrading to 0.5.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)
- Other changes required by [snowplow-web version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/#upgrading-to-0140), in particular the deduplication logic for Redshift and Postgres users

### Upgrading to 0.4.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Mobile
> Migration guide for upgrading the legacy Snowplow Mobile dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/mobile/

This is a legacy package.

### Upgrading to 0.7.0

- Version 1.4.0 of `dbt-core` now required

- You must add the following to the top level of your project yaml

  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```

- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

- **Redshift/Postgres users**: The deduplication logic has changed as a result of which any downstream joins on context and custom event tables will have to be deduped. Previously duplicate events were removed right in the base module to avoid any complications but this meant that the data could have been incomplete unlike in case of other data warehouses. These events will now arrive complete and deduped but they will likely have duplicate counterparts in their respective context tables as well, hence the warning.

  In case you have any custom models, please check them and make sure that deduplication logic is applied on them when they are joined to the main events table. We have provided a macro - `get_sde_or_context()` - for you to use for this purpose in the latest v0.14.0 snowplow-utils package. Check out the [package documentation](https://snowplow.github.io/dbt-snowplow-utils/#!/overview/snowplow_utils) on how to use it.

### Upgrading to 0.6.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Normalize
> Migration guide for upgrading the Snowplow Normalize dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/normalize/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- For a full upgrade walkthrough, please follow the [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.3.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

### Upgrading to 0.2.0

- Version 1.3.0 of `dbt-core` now required

- Upgrading your config file

  - Change the `event_name` field to `event_names` and make the value a list
  - Change the `self_describing_event_schema` field to `self_describing_event_schemas` and make the value a list
  - If you wish to make use of the new features, see the example config or the docs for more information

- Upgrading your models (preferred method is to re-run the python script, but can be done manually following these steps)

  - For each normalized model:

    - Convert the `event_name` and `sde_cols` fields to lists, and pluralize the names in both the set and the macro call
    - Add a new field, `sde_aliases` which is an empty list, add this between `sde_types` and `context_cols` in the macro call

  - For your filtered events table:

    - Change the `unique_key` in the config section to `unique_id`
    - Add a line between the `event_table_name` and from lines for each select statement; `, event_id||'-'||'<that_event_table_name>' as unique_id`, with the event table name for that select block.

  - For your users table:
    - Add 3 new values to the start of the macro call, `'user_id','',''`, before the `user_cols` argument.

- Upgrade your filtered events table
  - If you use the master filtered events table, you will need to add a new column for the latest version to work. If you have not processed much data yet it may be easier to simply re-run the package from scratch using dbt run --full-refresh --vars 'snowplow\_\_allow\_refresh: true', alternatively run the following in your warehouse, replacing the schema/dataset/warehouse and table name for your table:
    ```sql
    ALTER TABLE {schema}.{table} ADD COLUMN unique_id STRING;

    UPDATE {schema}.{table} SET unique_id = event_id||'-'||event_table_name WHERE 1 = 1;
    ```

---

# Migration guide for Unified Digital
> Migration guide for upgrading the Snowplow Unified Digital dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- for a full upgrade walkthrough, please follow [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.4.0

If you make use of the `conversion` module, and don't wish to do a full refresh, you will need to add the new column and update the existing records with the correct values for the model to run successfully incrementally.

**Redshift:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id TEXT;



UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = md5(cast(coalesce(cast(event_id as TEXT), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as TEXT), '_dbt_utils_surrogate_key_null_') as TEXT))

where 1=1;
```

**Snowflake:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id TEXT;



UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = md5(cast(coalesce(cast(event_id as TEXT), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as TEXT), '_dbt_utils_surrogate_key_null_') as TEXT))

where 1=1;
```

**BigQuery:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id string;



UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = to_hex(md5(cast(coalesce(cast(event_id as string), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as string), '_dbt_utils_surrogate_key_null_') as string)))

where 1=1;
```

**Databricks:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id string;



UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = md5(cast(coalesce(cast(event_id as string), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as string), '_dbt_utils_surrogate_key_null_') as string))

where 1=1;
```

***

### Upgrading to 0.3.0

- `snowplow__page_view_passthroughs` has been renamed to `snowplow__view_passthroughs`. Please rename this in your project yaml if you have set any.
- If you currently use `snowplow__allow_refresh` without also adding a `--full-refresh` flag in any script this is now required to ensure the manifest tables are fully refreshed.

### Upgrading to 0.2.0

The views table needs to be altered for existing Snowflake, Databricks or Redshift users as the data type of `engaged_time_in_s`, `absolute_time_in_s` have changed. Please modify the below script to fit your schemas and apply them before running the upgraded package.

The other option is to do a [complete refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.

#### Updates to the views table

- Changed type: `engaged_time_in_s`, `absolute_time_in_s`

**SQL scripts**

**snowflake:**

```sql
create or replace TRANSIENT TABLE (your_schema)_derived.snowplow_unified_views_new cluster by (to_date(start_tstamp))(

	VIEW_ID VARCHAR(16777216),

	EVENT_NAME VARCHAR(16777216),

	EVENT_ID VARCHAR(16777216),

	SESSION_IDENTIFIER VARCHAR(16777216),

	VIEW_IN_SESSION_INDEX NUMBER(18,0),

	VIEWS_IN_SESSION NUMBER(18,0),

	SESSION__PREVIOUS_SESSION_ID VARCHAR(36),

	USER_ID VARCHAR(16777216),

	USER_IDENTIFIER VARCHAR(16777216),

	STITCHED_USER_ID VARCHAR(16777216),

	NETWORK_USERID VARCHAR(16777216),

	DVCE_CREATED_TSTAMP TIMESTAMP_NTZ(9),

	COLLECTOR_TSTAMP TIMESTAMP_NTZ(9),

	DERIVED_TSTAMP TIMESTAMP_NTZ(9),

	START_TSTAMP TIMESTAMP_NTZ(9),

	END_TSTAMP TIMESTAMP_NTZ(9),

	MODEL_TSTAMP TIMESTAMP_NTZ(9),

	APP_ID VARCHAR(16777216),

	PLATFORM VARCHAR(16777216),

	DEVICE_IDENTIFIER VARCHAR(16777216),

	DEVICE_CATEGORY VARCHAR(16777216),

	DEVICE_SESSION_INDEX NUMBER(38,0),

	OS_VERSION VARCHAR(16777216),

	OS_TYPE VARCHAR(16777216),

	MOBILE__DEVICE_MANUFACTURER VARCHAR(16777216),

	MOBILE__DEVICE_MODEL VARCHAR(16777216),

	MOBILE__OS_TYPE VARCHAR(16777216),

	MOBILE__OS_VERSION VARCHAR(16777216),

	MOBILE__ANDROID_IDFA VARCHAR(16777216),

	MOBILE__APPLE_IDFA VARCHAR(16777216),

	MOBILE__APPLE_IDFV VARCHAR(16777216),

	MOBILE__CARRIER VARCHAR(16777216),

	MOBILE__OPEN_IDFA VARCHAR(16777216),

	MOBILE__NETWORK_TECHNOLOGY VARCHAR(16777216),

	MOBILE__NETWORK_TYPE VARCHAR(255),

	MOBILE__PHYSICAL_MEMORY NUMBER(38,0),

	MOBILE__SYSTEM_AVAILABLE_MEMORY NUMBER(38,0),

	MOBILE__APP_AVAILABLE_MEMORY NUMBER(38,0),

	MOBILE__BATTERY_LEVEL NUMBER(38,0),

	MOBILE__BATTERY_STATE VARCHAR(16777216),

	MOBILE__LOW_POWER_MODE BOOLEAN,

	MOBILE__AVAILABLE_STORAGE NUMBER(38,0),

	MOBILE__TOTAL_STORAGE NUMBER(38,0),

	MOBILE__IS_PORTRAIT BOOLEAN,

	MOBILE__RESOLUTION VARCHAR(16777216),

	MOBILE__SCALE FLOAT,

	MOBILE__LANGUAGE VARCHAR(16777216),

	MOBILE__APP_SET_ID VARCHAR(16777216),

	MOBILE__APP_SET_ID_SCOPE VARCHAR(16777216),

	OS_TIMEZONE VARCHAR(16777216),

	SCREEN_RESOLUTION VARCHAR(16777216),

	YAUAA__DEVICE_CLASS VARCHAR(16777216),

	YAUAA__DEVICE_VERSION VARCHAR(16777216),

	YAUAA__OPERATING_SYSTEM_VERSION VARCHAR(16777216),

	YAUAA__OPERATING_SYSTEM_CLASS VARCHAR(16777216),

	YAUAA__OPERATING_SYSTEM_NAME VARCHAR(16777216),

	YAUAA__OPERATING_SYSTEM_NAME_VERSION VARCHAR(16777216),

	GEO_COUNTRY VARCHAR(16777216),

	GEO_REGION VARCHAR(16777216),

	GEO_REGION_NAME VARCHAR(16777216),

	GEO_CITY VARCHAR(16777216),

	GEO_ZIPCODE VARCHAR(16777216),

	GEO_LATITUDE FLOAT,

	GEO_LONGITUDE FLOAT,

	GEO_TIMEZONE VARCHAR(16777216),

	USER_IPADDRESS VARCHAR(16777216),

	ENGAGED_TIME_IN_S DOUBLE,

	ABSOLUTE_TIME_IN_S DOUBLE,

	HORIZONTAL_PIXELS_SCROLLED NUMBER(38,0),

	VERTICAL_PIXELS_SCROLLED NUMBER(38,0),

	HORIZONTAL_PERCENTAGE_SCROLLED FLOAT,

	VERTICAL_PERCENTAGE_SCROLLED FLOAT,

	MKT_MEDIUM VARCHAR(16777216),

	MKT_SOURCE VARCHAR(16777216),

	MKT_TERM VARCHAR(16777216),

	MKT_CONTENT VARCHAR(16777216),

	MKT_CAMPAIGN VARCHAR(16777216),

	MKT_CLICKID VARCHAR(16777216),

	MKT_NETWORK VARCHAR(16777216),

	DEFAULT_CHANNEL_GROUP VARCHAR(25),

	PAGE_URL VARCHAR(16777216),

	PAGE_REFERRER VARCHAR(16777216),

	REFR_MEDIUM VARCHAR(16777216),

	REFR_SOURCE VARCHAR(16777216),

	REFR_TERM VARCHAR(16777216),

	PAGE_TITLE VARCHAR(16777216),

	CONTENT_GROUP VARCHAR(39),

	PAGE_URLSCHEME VARCHAR(16777216),

	PAGE_URLHOST VARCHAR(16777216),

	PAGE_URLPATH VARCHAR(16777216),

	PAGE_URLQUERY VARCHAR(16777216),

	PAGE_URLFRAGMENT VARCHAR(16777216),

	REFR_URLSCHEME VARCHAR(16777216),

	REFR_URLHOST VARCHAR(16777216),

	REFR_URLPATH VARCHAR(16777216),

	REFR_URLQUERY VARCHAR(16777216),

	REFR_URLFRAGMENT VARCHAR(16777216),

	BR_LANG VARCHAR(16777216),

	BR_VIEWWIDTH NUMBER(38,0),

	BR_VIEWHEIGHT NUMBER(38,0),

	BR_COLORDEPTH VARCHAR(16777216),

	BR_RENDERENGINE VARCHAR(16777216),

	DOC_WIDTH NUMBER(38,0),

	DOC_HEIGHT NUMBER(38,0),

	IAB__CATEGORY VARCHAR(16777216),

	IAB__PRIMARY_IMPACT VARCHAR(16777216),

	IAB__REASON VARCHAR(16777216),

	IAB__SPIDER_OR_ROBOT BOOLEAN,

	YAUAA__DEVICE_NAME VARCHAR(16777216),

	YAUAA__AGENT_CLASS VARCHAR(16777216),

	YAUAA__AGENT_NAME VARCHAR(16777216),

	YAUAA__AGENT_NAME_VERSION VARCHAR(16777216),

	YAUAA__AGENT_NAME_VERSION_MAJOR VARCHAR(16777216),

	YAUAA__AGENT_VERSION VARCHAR(16777216),

	YAUAA__AGENT_VERSION_MAJOR VARCHAR(16777216),

	YAUAA__LAYOUT_ENGINE_CLASS VARCHAR(16777216),

	YAUAA__LAYOUT_ENGINE_NAME VARCHAR(16777216),

	YAUAA__LAYOUT_ENGINE_NAME_VERSION VARCHAR(16777216),

	YAUAA__LAYOUT_ENGINE_NAME_VERSION_MAJOR VARCHAR(16777216),

	YAUAA__LAYOUT_ENGINE_VERSION VARCHAR(16777216),

	YAUAA__LAYOUT_ENGINE_VERSION_MAJOR VARCHAR(16777216),

	UA__DEVICE_FAMILY VARCHAR(16777216),

	UA__OS_VERSION VARCHAR(16777216),

	UA__OS_MAJOR VARCHAR(16777216),

	UA__OS_MINOR VARCHAR(16777216),

	UA__OS_PATCH VARCHAR(16777216),

	UA__OS_PATCH_MINOR VARCHAR(16777216),

	UA__USERAGENT_FAMILY VARCHAR(16777216),

	UA__USERAGENT_MAJOR VARCHAR(16777216),

	UA__USERAGENT_MINOR VARCHAR(16777216),

	UA__USERAGENT_PATCH VARCHAR(16777216),

	UA__USERAGENT_VERSION VARCHAR(16777216),

	SCREEN_VIEW__NAME VARCHAR(16777216),

	SCREEN_VIEW__PREVIOUS_ID VARCHAR(36),

	SCREEN_VIEW__PREVIOUS_NAME VARCHAR(16777216),

	SCREEN_VIEW__PREVIOUS_TYPE VARCHAR(16777216),

	SCREEN_VIEW__TRANSITION_TYPE VARCHAR(16777216),

	SCREEN_VIEW__TYPE VARCHAR(16777216),

	APP__BUILD VARCHAR(255),

	APP__VERSION VARCHAR(255),

	GEO__ALTITUDE FLOAT,

	GEO__ALTITUDE_ACCURACY FLOAT,

	GEO__BEARING FLOAT,

	GEO__LATITUDE FLOAT,

	GEO__LATITUDE_LONGITUDE_ACCURACY FLOAT,

	GEO__LONGITUDE FLOAT,

	GEO__SPEED FLOAT,

	SCREEN__FRAGMENT VARCHAR(16777216),

	SCREEN__TOP_VIEW_CONTROLLER VARCHAR(16777216),

	SCREEN__VIEW_CONTROLLER VARCHAR(16777216),

	USERAGENT VARCHAR(16777216),

	V_COLLECTOR VARCHAR(16777216),

	EVENT_ID2 VARCHAR(16777216),

	LAST_LIST_ITEM_INDEX NUMBER(38,0),

	LIST_ITEMS_COUNT NUMBER(38,0),

	LIST_ITEMS_PERCENTAGE_SCROLLED FLOAT

);



insert into (your_schema)_derived.snowplow_unified_views_new  select *, NULL AS LAST_LIST_ITEM_INDEX, NULL AS LIST_ITEMS_COUNT, NULL AS LIST_ITEMS_PERCENTAGE_SCROLLED from (your_schema)_derived.snowplow_unified_views;

drop table (your_schema)_derived.snowplow_unified_views;

create table (your_schema)_derived.snowplow_unified_views as select * from (your_schema)_derived.snowplow_unified_views_new;

drop table (your_schema)_derived.snowplow_unified_views_new;
```

**databricks:**

#### Updates to the views table

```sql
CREATE TABLE (your_schema)_derived.snowplow_unified_views_new (

  view_id STRING,

  event_name STRING,

  event_id STRING,

  session_identifier STRING,

  view_in_session_index INT,

  views_in_session INT,

  session__previous_session_id STRING,

  user_id STRING,

  user_identifier STRING,

  stitched_user_id STRING,

  network_userid STRING,

  dvce_created_tstamp TIMESTAMP,

  collector_tstamp TIMESTAMP,

  derived_tstamp TIMESTAMP,

  start_tstamp TIMESTAMP,

  end_tstamp TIMESTAMP,

  model_tstamp TIMESTAMP,

  app_id STRING,

  platform STRING,

  device_identifier STRING,

  device_category STRING,

  device_session_index INT,

  os_version STRING,

  os_type STRING,

  mobile__device_manufacturer STRING,

  mobile__device_model STRING,

  mobile__os_type STRING,

  mobile__os_version STRING,

  mobile__android_idfa STRING,

  mobile__apple_idfa STRING,

  mobile__apple_idfv STRING,

  mobile__carrier STRING,

  mobile__open_idfa STRING,

  mobile__network_technology STRING,

  mobile__network_type STRING,

  mobile__physical_memory INT,

  mobile__system_available_memory INT,

  mobile__app_available_memory INT,

  mobile__battery_level INT,

  mobile__battery_state STRING,

  mobile__low_power_mode BOOLEAN,

  mobile__available_storage BIGINT,

  mobile__total_storage BIGINT,

  mobile__is_portrait BOOLEAN,

  mobile__resolution STRING,

  mobile__scale FLOAT,

  mobile__language STRING,

  mobile__app_set_id STRING,

  mobile__app_set_id_scope STRING,

  screen_resolution STRING,

  geo_country STRING,

  geo_region BIGINT,

  geo_region_name BIGINT,

  geo_city BIGINT,

  geo_zipcode BIGINT,

  geo_latitude BIGINT,

  geo_longitude BIGINT,

  geo_timezone BIGINT,

  user_ipaddress STRING,

  engaged_time_in_s FLOAT,

  absolute_time_in_s FLOAT,

  horizontal_pixels_scrolled BIGINT,

  vertical_pixels_scrolled BIGINT,

  horizontal_percentage_scrolled FLOAT,

  vertical_percentage_scrolled FLOAT,

  last_list_item_index INT,

  list_items_count INT,

  list_items_percentage_scrolled FLOAT,

  mkt_medium STRING,

  mkt_source STRING,

  mkt_term STRING,

  mkt_content STRING,

  mkt_campaign STRING,

  mkt_clickid BIGINT,

  mkt_network BIGINT,

  default_channel_group STRING,

  page_url STRING,

  page_referrer STRING,

  refr_medium STRING,

  refr_source STRING,

  refr_term STRING,

  screen_view__name STRING,

  screen_view__previous_id STRING,

  screen_view__previous_name STRING,

  screen_view__previous_type STRING,

  screen_view__transition_type STRING,

  screen_view__type STRING,

  app__build STRING,

  app__version STRING,

  screen__fragment STRING,

  screen__top_view_controller STRING,

  screen__view_controller STRING,

  useragent STRING,

  v_collector STRING,

  event_id2 STRING,

  start_tstamp_date DATE)

USING delta

PARTITIONED BY (start_tstamp_date)

TBLPROPERTIES (

  'delta.autoOptimize.autoCompact' = 'true',

  'delta.autoOptimize.optimizeWrite' = 'true',

  'delta.checkpoint.writeStatsAsJson' = 'false',

  'delta.checkpoint.writeStatsAsStruct' = 'true',

  'delta.minReaderVersion' = '1',

  'delta.minWriterVersion' = '2');



insert into (your_schema)_derived.snowplow_unified_views_new  select *, null as last_list_item_index, null as list_items_count, null as list_items_percentage_scrolled from (your_schema)_derived.snowplow_unified_views;

drop table (your_schema)_derived.snowplow_unified_views;

create table (your_schema)_derived.snowplow_unified_views as select * from (your_schema)_derived.snowplow_unified_views_new;

drop table (your_schema)_derived.snowplow_unified_views_new;
```

**redshift:**

#### Updates to the views table

```sql
CREATE TABLE IF NOT EXISTS (your_schema)_derived.snowplow_unified_views_new

(

	view_id VARCHAR(65535)   ENCODE lzo

	,event_name VARCHAR(22)   ENCODE lzo

	,event_id VARCHAR(36)   ENCODE lzo

	,session_identifier VARCHAR(36)   ENCODE lzo

	,view_in_session_index BIGINT   ENCODE az64

	,views_in_session BIGINT   ENCODE az64

	,session__previous_session_id VARCHAR(36)   ENCODE lzo

	,user_id VARCHAR(256)   ENCODE lzo

	,user_identifier VARCHAR(36)   ENCODE lzo

	,stitched_user_id VARCHAR(65535)   ENCODE lzo

	,network_userid VARCHAR(36)   ENCODE lzo

	,dvce_created_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

	,collector_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

	,derived_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

	,start_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE RAW

	,end_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

	,model_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64

	,app_id VARCHAR(256)   ENCODE lzo

	,platform VARCHAR(3)   ENCODE lzo

	,device_identifier VARCHAR(36)   ENCODE lzo

	,device_category VARCHAR(22)   ENCODE lzo

	,device_session_index INTEGER   ENCODE az64

	,os_version VARCHAR(3)   ENCODE lzo

	,os_type VARCHAR(25)   ENCODE lzo

	,mobile__device_manufacturer VARCHAR(7)   ENCODE lzo

	,mobile__device_model VARCHAR(37)   ENCODE lzo

	,mobile__os_type VARCHAR(25)   ENCODE lzo

	,mobile__os_version VARCHAR(3)   ENCODE lzo

	,mobile__android_idfa VARCHAR(36)   ENCODE lzo

	,mobile__apple_idfa VARCHAR(2)   ENCODE lzo

	,mobile__apple_idfv VARCHAR(2)   ENCODE lzo

	,mobile__carrier VARCHAR(7)   ENCODE lzo

	,mobile__open_idfa VARCHAR(2)   ENCODE lzo

	,mobile__network_technology VARCHAR(2)   ENCODE lzo

	,mobile__network_type VARCHAR(4)   ENCODE lzo

	,mobile__physical_memory INTEGER   ENCODE az64

	,mobile__system_available_memory INTEGER   ENCODE az64

	,mobile__app_available_memory INTEGER   ENCODE az64

	,mobile__battery_level INTEGER   ENCODE az64

	,mobile__battery_state VARCHAR(8)   ENCODE lzo

	,mobile__low_power_mode BOOLEAN   ENCODE RAW

	,mobile__available_storage INTEGER   ENCODE az64

	,mobile__total_storage BIGINT   ENCODE az64

	,mobile__is_portrait BOOLEAN   ENCODE RAW

	,mobile__resolution VARCHAR(9)   ENCODE lzo

	,mobile__scale DOUBLE PRECISION   ENCODE RAW

	,mobile__language VARCHAR(2)   ENCODE lzo

	,mobile__app_set_id VARCHAR(2)   ENCODE lzo

	,mobile__app_set_id_scope VARCHAR(2)   ENCODE lzo

	,screen_resolution VARCHAR(9)   ENCODE lzo

	,geo_country VARCHAR(256)   ENCODE lzo

	,geo_region INTEGER   ENCODE az64

	,geo_region_name INTEGER   ENCODE az64

	,geo_city INTEGER   ENCODE az64

	,geo_zipcode INTEGER   ENCODE az64

	,geo_latitude INTEGER   ENCODE az64

	,geo_longitude INTEGER   ENCODE az64

	,geo_timezone INTEGER   ENCODE az64

	,user_ipaddress VARCHAR(10)   ENCODE lzo

	,engaged_time_in_s DOUBLE PRECISION   ENCODE RAW

	,absolute_time_in_s DOUBLE PRECISION   ENCODE RAW

	,horizontal_pixels_scrolled INTEGER   ENCODE az64

	,vertical_pixels_scrolled INTEGER   ENCODE az64

	,horizontal_percentage_scrolled DOUBLE PRECISION   ENCODE RAW

	,vertical_percentage_scrolled DOUBLE PRECISION   ENCODE RAW

	,last_list_item_index INTEGER   ENCODE az64

	,list_items_count INTEGER   ENCODE az64

	,list_items_percentage_scrolled DOUBLE PRECISION   ENCODE RAW

	,mkt_medium VARCHAR(256)   ENCODE lzo

	,mkt_source VARCHAR(256)   ENCODE lzo

	,mkt_term VARCHAR(256)   ENCODE lzo

	,mkt_content VARCHAR(256)   ENCODE lzo

	,mkt_campaign VARCHAR(256)   ENCODE lzo

	,mkt_clickid INTEGER   ENCODE az64

	,mkt_network INTEGER   ENCODE az64

	,default_channel_group VARCHAR(25)   ENCODE lzo

	,page_url VARCHAR(256)   ENCODE lzo

	,page_referrer VARCHAR(256)   ENCODE lzo

	,refr_medium VARCHAR(256)   ENCODE lzo

	,refr_source VARCHAR(256)   ENCODE lzo

	,refr_term VARCHAR(256)   ENCODE lzo

	,screen_view__name VARCHAR(2)   ENCODE lzo

	,screen_view__previous_id VARCHAR(2)   ENCODE lzo

	,screen_view__previous_name VARCHAR(2)   ENCODE lzo

	,screen_view__previous_type VARCHAR(2)   ENCODE lzo

	,screen_view__transition_type VARCHAR(2)   ENCODE lzo

	,screen_view__type VARCHAR(2)   ENCODE lzo

	,app__build VARCHAR(2)   ENCODE lzo

	,app__version VARCHAR(2)   ENCODE lzo

	,screen__fragment VARCHAR(2)   ENCODE lzo

	,screen__top_view_controller VARCHAR(2)   ENCODE lzo

	,screen__view_controller VARCHAR(2)   ENCODE lzo

	,useragent VARCHAR(51)   ENCODE lzo

	,v_collector VARCHAR(28)   ENCODE lzo

	,event_id2 VARCHAR(36)   ENCODE lzo

)

DISTSTYLE KEY

 DISTKEY (view_id)

 SORTKEY (

	start_tstamp

	)

;



insert into (your_schema)_derived.snowplow_unified_views_new  select *, null as last_list_item_index, null as list_items_count, null as list_items_percentage_scrolled from (your_schema)_derived.snowplow_unified_views;

drop table (your_schema)_derived.snowplow_unified_views;

alter table (your_schema)_derived.snowplow_unified_views_new

rename to snowplow_unified_views;
```

***

---

# Migration guide for Utils
> Migration guide for upgrading the Snowplow Utils dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- for a full upgrade walkthrough, please follow [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.14.0

- Version 1.4.0 of `dbt-core` now required
- Any custom models using the `snowplow_incremental` materialization should be changed to use the `incremental` materialization with `snowplow_optimize=true` set in the model config. `snowplow_incremental` has been deprecated
- BigQuery incremental models now require the `upsert_date_key` config
- `snowplow_utils.type_string` it has been deprecated and should be replaced with `dbt.type_string()` for non-length specific calls, and `snowplow_utils.type_max_string` for redshift strings above 256 characters.
- `snowplow_utils.get_cluster_by` and `snowplow_utils.get_partition_by` they have been deprecated and should be replaced with `snowplow_utils.get_value_by_target_type` with arg names `bigquery_val`, `databricks_val` etc.
- `snowplow_utils.snowplow_is_incremental` has been deprecated and should be replaced with standard `is_incremental()` to align with removal of the custom materialization
- You must add the following to the top level of your project yaml to take advantage of our optimized incremental materialization
  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```

### Upgrading to 0.13.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Web
> Migration guide for upgrading the legacy Snowplow Web dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/

This is a legacy package.

### Upgrading to 0.16.0

Note the minimum `dbt-core` version now required is 1.5.0

Most upgrades and new features should be available automatically after the first run of the new version of the package, however there are a few breaking changes to manage before this first run. To enable any other new optional features or make use of the new configuration options, please see the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/web/) page.

1. The seed files have been prefixed with `snowplow_web_` so you will need to re-run `dbt seed` if this is not part of your standard workflow (note this will not drop the existing tables)
2. Both the quarantined sessions and the sessions lifecycle table have has column(s) renamed, please run the below statements to rename this column in your warehouse if not doing a full-refresh of the package.
   ```sql
   alter table <your_schema>_snowplow_manifest.snowplow_web_base_sessions_lifecycle_manifest rename column session_id to session_identifier;

   alter table <your_schema>_snowplow_manifest.snowplow_web_base_quarantined_sessions rename column session_id to session_identifier;

   alter table <your_schema>_snowplow_manifest.snowplow_web_base_sessions_lifecycle_manifest rename column domain_userid to user_identifier;
   ```
3. For Redshift/Postgres users, the context variables are now just the table name, no need for the complicated `source` function if you had overwritten these previously.

### Upgrading to 0.15.1

For existing Snowflake and Databricks Core Web Vitals optional module users the derived table `snowplow_web_vitals` needs to be altered due to column data type change. Please run the below queries to migrate. The other option is to do a [complete refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.

**SQL scripts**

**snowflake:**

```sql
create table (your_schema)_derived.snowplow_web_vitals_new

(



    event_id               STRING,

    event_name             STRING,

    app_id                 STRING,

    platform               STRING,

    domain_userid          STRING,

    user_id                STRING,

    page_view_id           STRING,

    domain_sessionid       STRING,

    collector_tstamp       TIMESTAMP,

    derived_tstamp         TIMESTAMP,

    load_tstamp            TIMESTAMP,

    geo_country            STRING,

    page_url               STRING,

    url_group              STRING,

    page_title             STRING,

    useragent              STRING,

    device_class           STRING,

    device_name            STRING,

    agent_name             STRING,

    agent_version          STRING,

    operating_system_name  STRING,

    lcp                    DECIMAL(14, 4),

    fcp                    DECIMAL(14, 4),

    fid                    DECIMAL(14, 4),

    cls                    DECIMAL(14, 4),

    inp                    DECIMAL(14, 4),

    ttfb                   DECIMAL(14, 4),

    navigation_type        STRING,

    dedupe_index           INT,

    lcp_result             STRING,

    fid_result             STRING,

    cls_result             STRING,

    ttfb_result            STRING,

    inp_result             STRING

);



insert into (your_schema)_derived.snowplow_web_vitals_new  select * from (your_schema)_derived.snowplow_web_vitals;

drop table (your_schema)_derived.snowplow_web_vitals;

alter table (your_schema)_derived.snowplow_web_vitals_new rename to (your_schema)_derived.snowplow_web_vitals;
```

**databricks:**

```sql
create table (your_schema)_derived.snowplow_web_vitals_new

(

    event_id               STRING,

    event_name             STRING,

    app_id                 STRING,

    platform               STRING,

    domain_userid          STRING,

    user_id                STRING,

    page_view_id           STRING,

    domain_sessionid       STRING,

    collector_tstamp       TIMESTAMP,

    derived_tstamp         TIMESTAMP,

    load_tstamp            TIMESTAMP,

    geo_country            STRING,

    page_url               STRING,

    url_group              STRING,

    page_title             STRING,

    useragent              STRING,

    device_class           STRING,

    device_name            STRING,

    agent_name             STRING,

    agent_version          STRING,

    operating_system_name  STRING,

    lcp                    DECIMAL(14, 4),

    fcp                    DECIMAL(14, 4),

    fid                    DECIMAL(14, 4),

    cls                    DECIMAL(14, 4),

    inp                    DECIMAL(14, 4),

    ttfb                   DECIMAL(14, 4),

    navigation_type        STRING,

    dedupe_index           INT,

    lcp_result             STRING,

    fid_result             STRING,

    cls_result             STRING,

    ttfb_result            STRING,

    inp_result             STRING

);



insert into (your_schema)_derived.snowplow_web_vitals_new  select * from (your_schema)_derived.snowplow_web_vitals;

drop table (your_schema)_derived.snowplow_web_vitals;

create table (your_schema)_derived.snowplow_web_vitals select * from (your_schema)_derived.snowplow_web_vitals_new;

drop table (your_schema)_derived.snowplow_web_vitals_new;
```

***

### Upgrading to 0.15.0

Some of the new fields that are added will contain nulls for existing records. To avoid failing out of the box tests please update your profiles.yml to disable those tests:

```yml
# dbt_project.yml

...



tests:

  snowplow_web:

    sessions:

      not_null_snowplow_web_sessions_is_engaged:

        +enabled: false

      not_null_snowplow_web_sessions_total_events:

        +enabled: false
```

- Consent enablement is now done via the `snowplow__enable_consent` variable instead of via the enable block. If you are using consent please set this variable to `true`
- Some seed files are now required to run the package, please run `dbt seed --select snowplow_web --full-refresh` to load these. The schema they are loaded to can be changed in your `dbt_project.yml`
- New columns will be auto-added to users, sessions, and page views tables, please be aware for any downstream queries/BI tools.

### Upgrading to 0.14.0

- Version 1.4.0 of `dbt-core` now required

- You must add the following to the top level of your project yaml

  ```yml
  dispatch:

    - macro_namespace: dbt

      search_order: ['snowplow_utils', 'dbt']
  ```

- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

- **Redshift/Postgres users**: The deduplication logic has changed as a result of which any downstream joins on context and custom event tables will have to be deduped. Previously duplicate events were removed right in the base module to avoid any complications but this meant that the data could have been incomplete unlike in case of other data warehouses. These events will now arrive complete and deduped but they will likely have duplicate counterparts in their respective context tables as well, hence the warning.

  In case you have any custom models, please check them and make sure that deduplication logic is applied on them when they are joined to the main events table. We have provided a macro - `get_sde_or_context()` - for you to use for this purpose in the latest v0.14.0 snowplow-utils package. Check out the [package documentation](https://snowplow.github.io/dbt-snowplow-utils/#!/overview/snowplow_utils) on how to use it.

### Upgrading to 0.13.0

- RDB Loader 4.0.0 or BigQuery Loader 1.0.0. If using postgres loader or older versions set `snowplow__enable_load_tstamp` to `false` in your project yaml and you will not be able to use the consent models.

### Upgrading to 0.12.0

- Version 1.3.0 of `dbt-core` now required

---

# Migrate from Web to Unified Digital
> Migration guide for transitioning from the legacy Snowplow Web dbt package to the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web_to_unified/

Follow this migration guide to upgrade from the legacy [Snowplow Web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) dbt package to the [Snowplow Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) package. Unified Digital is the most supported and feature-rich Snowplow dbt package, which you can use to model web and/or mobile event data.

## Simplified upgrade overview

![Migration steps - part I](/assets/images/migrate_1-d38ec8820d5eb2d61f1ef4c7633f225a.png)

![Migration steps - part II](/assets/images/migrate_2-4382295f0ebf3e74071762d3f2ec5b77.png)

## Breaking changes between the packages

- The `page_views` table has been renamed to `views`
- The `domain_userid` field has been renamed to `user_identifier`, as you now have the ability to define your own custom logic / alternative user id field for the aggregation
- The `domain_sessionid` has been renamed to `session_identifier`, as you now have the ability to define your own custom logic / alternative session id field for the aggregation
- Apart from these, fields generated by the out-of-the-box entities and enrichments have been added a prefix to indicate where they are coming from, e.g. `device_version` has been renamed to `yauaa__device_version`
- These out-of-the-box entities and enrichments can be optionally enabled or disabled for the model (previously these values were there as NULLs) through variables e.g. `snowplow__enable_yauaa: true` which are then flattened automatically within the `base_events_this_run` table and can be referred to from there directly for custom models, if needed
- Some package variables have been renamed, as well as new ones added

## Things to bear in mind

Due to the breaking changes, ideally it's best to start from scratch and run the new package on your whole historic events dataset. This could be costly in both time and effort. In certain scenarios, e.g. in case of large data volumes, it may be useful to try and upgrade instead using the existing modeled derived dataset.

There is no 100% accurate upgrade solution, but with this guide you may be able to get 95% of the way there.

1. **Incomplete or inaccurate fields for historic data**: the unified package contains a list of new fields, some of which we can compute on a row-by-row basis, in which case we provided update scripts to compute them for your existing derived data. There are, however, a few other new fields that are a result of aggregate or other computation that we cannot calculate without a full refresh. E.g. `absolute_time_in_s` for users therefore they will stay `NULL`. Then when a new event comes to update that record it may either correct it or it will just be inaccurate.

2. **Fields that aren't available in the out-of-the-box derived tables**: there are a few fields that we've dropped as they were incorporated into a new field based on different grouping. There may also be custom fields you have added to the derived tables. The below passthrough variables come in handy in this case:

```yml
    snowplow__view_passthroughs: []

    snowplow__session_passthroughs: []

    snowplow__user_first_passthroughs: []

    snowplow__user_last_passthroughs: []
```

For more on this, have a look at the [passthrough fields](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) section.

## Upgrade steps

### Step 1: Migrate to the latest version of the Web package

Once you decided you would like to go ahead with the upgrade process you will first need to make sure your version of the web model is version 1.0.0 or version 1.0.1. The [Web model migration guides](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/) will help you decide what other changes you need to take into consideration. If the changes apply to your warehouse, you may need to execute the upgrade SQL scripts to bridge the gap.

### Step 2: Execute SQL scripts to create the new tables

The provided SQL scripts will first create the new tables based on your existing derived tables created by the web package. Secondly, they will make the changes i.e. renaming, adding, dropping, and updating columns wherever possible, including in the manifest tables. You will then have everything ready for a new run in the Unified Digital package as if nothing happened.

Update `(your_schema)` in the SQL scripts to match your derived schema name. Execute the scripts with your database IDE to create your new derived and manifest tables without having to reprocess your event data from scratch.

You might need to adjust the data `varchar` data type to `string` depending on your warehouse (most likely to be relevant for BigQuery or Databricks users) or limit it to the maximum in case there are limitations (potentially Redshift).

**SQL scripts**

```sql
create table (your_schema)_derived.snowplow_unified_views as (select * from (your_schema)_derived.snowplow_web_page_views);

create table (your_schema)_derived.snowplow_unified_sessions as (select * from (your_schema)_derived.snowplow_web_sessions);

create table (your_schema)_derived.snowplow_unified_users as (select * from (your_schema)_derived.snowplow_web_users);

create table (your_schema)_snowplow_manifest.snowplow_unified_incremental_manifest as (select * from (your_schema)_snowplow_manifest.snowplow_web_incremental_manifest);

create table (your_schema)_snowplow_manifest.snowplow_unified_base_sessions_lifecycle_manifest as (select * from (your_schema)_snowplow_manifest.snowplow_web_base_sessions_lifecycle_manifest)



alter table (your_schema)_derived.snowplow_unified_views rename column domain_sessionid to session_identifier;

alter table (your_schema)_derived.snowplow_unified_views rename column domain_userid to user_identifier;

alter table (your_schema)_derived.snowplow_unified_views rename column original_domain_userid to device_identifier;

alter table (your_schema)_derived.snowplow_unified_views drop column original_domain_sessionid;

alter table (your_schema)_derived.snowplow_unified_views rename column agent_class to yauaa__agent_class;

alter table (your_schema)_derived.snowplow_unified_views rename column agent_name to yauaa__agent_name;

alter table (your_schema)_derived.snowplow_unified_views rename column agent_name_version to yauaa__agent_name_version;

alter table (your_schema)_derived.snowplow_unified_views rename column agent_name_version_major to yauaa__agent_name_version_major;

alter table (your_schema)_derived.snowplow_unified_views rename column agent_version to yauaa__agent_version;

alter table (your_schema)_derived.snowplow_unified_views rename column agent_version_major to yauaa__agent_version_major;

alter table (your_schema)_derived.snowplow_unified_views rename column category to iab__category;

alter table (your_schema)_derived.snowplow_unified_views drop column device_brand;

alter table (your_schema)_derived.snowplow_unified_views rename column device_class to yauaa__device_class;

alter table (your_schema)_derived.snowplow_unified_views rename column device_family to ua__device_family;

alter table (your_schema)_derived.snowplow_unified_views rename column device_name to yauaa__device_name;

alter table (your_schema)_derived.snowplow_unified_views rename column device_version to yauaa__device_version;

alter table (your_schema)_derived.snowplow_unified_views rename column domain_sessionidx to device_session_index;

alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_class to yauaa__layout_engine_class;

alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_name to yauaa__layout_engine_name;

alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_name_version to yauaa__layout_engine_name_version;

alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_name_version_major to yauaa__layout_engine_name_version_major;

alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_version to yauaa__layout_engine_version;

alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_version_major to yauaa__layout_engine_version_major;

alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_class to yauaa__operating_system_class;

alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_name to yauaa__operating_system_name;

alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_name_version to yauaa__operating_system_name_version;

alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_version to yauaa__operating_system_version;

alter table  (your_schema)_derived.snowplow_unified_views drop column os_family;

alter table  (your_schema)_derived.snowplow_unified_views rename column os_major to ua__os_major;

alter table  (your_schema)_derived.snowplow_unified_views rename column os_minor to ua__os_minor;

alter table  (your_schema)_derived.snowplow_unified_views rename column os_patch to ua__os_patch;

alter table  (your_schema)_derived.snowplow_unified_views rename column os_patch_minor to ua__os_patch_minor;

alter table  (your_schema)_derived.snowplow_unified_views rename column os_version to ua__os_version;

alter table  (your_schema)_derived.snowplow_unified_views rename column page_views_in_session to views_in_session;

alter table  (your_schema)_derived.snowplow_unified_views rename column page_view_id to view_id;

alter table  (your_schema)_derived.snowplow_unified_views rename column page_view_in_session_index to view_in_session_index;

alter table  (your_schema)_derived.snowplow_unified_views rename column primary_impact to iab__primary_impact;

alter table  (your_schema)_derived.snowplow_unified_views rename column reason to iab__reason;

alter table  (your_schema)_derived.snowplow_unified_views rename column spider_or_robot to iab__spider_or_robot;

alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_family to ua__useragent_family;

alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_major to ua__useragent_major;

alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_minor to ua__useragent_minor;

alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_patch to ua__useragent_patch;

alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_version to ua__useragent_version;

alter table  (your_schema)_derived.snowplow_unified_views add column if not exists default_channel_group varchar(25);

alter table  (your_schema)_derived.snowplow_unified_views add column event_name varchar(1000);

update (your_schema)_derived.snowplow_unified_views

set event_name = 'page_view'

where 1=1;

alter table  (your_schema)_derived.snowplow_unified_views add column os_type varchar(16777216);

alter table  (your_schema)_derived.snowplow_unified_views add column os_version varchar(16777216);

alter table  (your_schema)_derived.snowplow_unified_views add column session__previous_session_id varchar(36);

alter table  (your_schema)_derived.snowplow_unified_views add column if not exists stitched_user_id varchar(16777216);



alter table (your_schema)_derived.snowplow_unified_sessions rename column domain_sessionid to session_identifier;

alter table (your_schema)_derived.snowplow_unified_sessions rename column domain_userid to user_identifier;

alter table (your_schema)_derived.snowplow_unified_sessions rename column original_domain_userid to device_identifier;

alter table (your_schema)_derived.snowplow_unified_sessions drop column original_domain_sessionid;

alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_class to yauaa__agent_class;

alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_name to yauaa__agent_name;

alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_name_version to yauaa__agent_name_version;

alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_name_version_major to yauaa__agent_name_version_major;

alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_version to yauaa__agent_version;

alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_version_major to yauaa__agent_version_major;

alter table (your_schema)_derived.snowplow_unified_sessions rename column br_lang to first_br_lang;

alter table (your_schema)_derived.snowplow_unified_sessions rename column br_lang_name to first_br_lang_name;

alter table (your_schema)_derived.snowplow_unified_sessions rename column category to iab__category;

alter table (your_schema)_derived.snowplow_unified_sessions drop column device_brand;

alter table (your_schema)_derived.snowplow_unified_sessions rename column device_class to yauaa__device_class;

alter table (your_schema)_derived.snowplow_unified_sessions rename column device_family to ua__device_family;

alter table (your_schema)_derived.snowplow_unified_sessions rename column device_name to yauaa__device_name;

alter table (your_schema)_derived.snowplow_unified_sessions rename column device_version to yauaa__device_version;

alter table (your_schema)_derived.snowplow_unified_sessions rename column domain_sessionidx to device_session_index;

alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_city to first_geo_city;

alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_continent to first_geo_continent;

alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_country to first_geo_country;

alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_country_name to first_geo_country_name;

alter table (your_schema)_derived.snowplow_unified_sessions drop column geo_region;

alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_region_name to first_geo_region_name;

alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_class to yauaa__layout_engine_class;

alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_name to yauaa__layout_engine_name;

alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_name_version to yauaa__layout_engine_name_version;

alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_name_version_major to yauaa__layout_engine_name_version_major;

alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_version to yauaa__layout_engine_version;

alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_version_major to yauaa__layout_engine_version_major;

alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_class to yauaa__operating_system_class;

alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_name to yauaa__operating_system_name;

alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_name_version to yauaa__operating_system_name_version;

alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_version to yauaa__operating_system_version;

alter table (your_schema)_derived.snowplow_unified_sessions drop column os_family;

alter table (your_schema)_derived.snowplow_unified_sessions rename column os_major to ua__os_major;

alter table (your_schema)_derived.snowplow_unified_sessions rename column os_minor to ua__os_minor;

alter table (your_schema)_derived.snowplow_unified_sessions rename column os_patch to ua__os_patch;

alter table (your_schema)_derived.snowplow_unified_sessions rename column os_patch_minor to ua__os_patch_minor;

alter table (your_schema)_derived.snowplow_unified_sessions rename column page_views to views;

alter table (your_schema)_derived.snowplow_unified_sessions rename column primary_impact to iab__primary_impact;

alter table (your_schema)_derived.snowplow_unified_sessions rename column reason to iab__reason;

alter table (your_schema)_derived.snowplow_unified_sessions rename column referrer to page_referrer;

alter table (your_schema)_derived.snowplow_unified_sessions rename column spider_or_robot to iab__spider_or_robot;

alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_family to ua__useragent_family;

alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_major to ua__useragent_major;

alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_minor to ua__useragent_minor;

alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_patch to ua__useragent_patch;

alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_version to ua__useragent_version;

alter table (your_schema)_derived.snowplow_unified_sessions add column os_type varchar(16777216);

alter table (your_schema)_derived.snowplow_unified_sessions add column session_duration_s number(18,0);

alter table (your_schema)_derived.snowplow_unified_sessions add column session__previous_session_id varchar(36);

alter table (your_schema)_derived.snowplow_unified_sessions add column first_event_name varchar(1000);

alter table (your_schema)_derived.snowplow_unified_sessions add column last_event_name varchar(1000);

alter table  (your_schema)_derived.snowplow_unified_sessions add column if not exists stitched_user_id varchar(16777216);



alter table (your_schema)_derived.snowplow_unified_users rename column domain_userid to user_identifier;

alter table (your_schema)_derived.snowplow_unified_users drop column original_domain_userid;

alter table (your_schema)_derived.snowplow_unified_users rename column page_views to views;

alter table (your_schema)_derived.snowplow_unified_users rename column referrer to page_referrer;

alter table (your_schema)_derived.snowplow_unified_users add column on_mobile boolean;

update (your_schema)_derived.snowplow_unified_users

set on_mobile = false

where 1=1;

alter table (your_schema)_derived.snowplow_unified_users add column on_web boolean;

update (your_schema)_derived.snowplow_unified_users

set on_web = true

where 1=1;

alter table (your_schema)_derived.snowplow_unified_users add column screen_names_viewed number(30,0);

alter table (your_schema)_derived.snowplow_unified_users add column sessions_duration_s number(30,0);

alter table (your_schema)_derived.snowplow_unified_users add column active_days number(18,0);

alter table (your_schema)_derived.snowplow_unified_users add column last_platform varchar(255);

update (your_schema)_derived.snowplow_unified_users

set last_platform = 'web'

where 1=1;

alter table (your_schema)_derived.snowplow_unified_users add column last_screen_resolution varchar(16777216);

alter table (your_schema)_derived.snowplow_unified_users add column last_os_type varchar(16777216);

alter table (your_schema)_derived.snowplow_unified_users add column last_os_version varchar(16777216);

alter table (your_schema)_derived.snowplow_unified_users add column first_platform varchar(255);

update (your_schema)_derived.snowplow_unified_users

set first_platform = 'web'

where 1=1;

alter table (your_schema)_derived.snowplow_unified_users add column geo_latitude float;

alter table (your_schema)_derived.snowplow_unified_users add column geo_longitude float;

alter table (your_schema)_derived.snowplow_unified_users add column geo_timezone varchar(64);

alter table (your_schema)_derived.snowplow_unified_users add column geo_zipcode varchar(15);

alter table  (your_schema)_derived.snowplow_unified_users add column if not exists stitched_user_id varchar(16777216);



update (your_schema)_snowplow_manifest.snowplow_unified_incremental_manifest

set model = replace(model, 'snowplow_web', 'snowplow_unified')

where 1=1;



update (your_schema)_snowplow_manifest.snowplow_unified_incremental_manifest

set model = case when model = 'snowplow_unified_page_views_this_run' then 'snowplow_unified_views_this_run'

when model = 'snowplow_unified_page_views' then 'snowplow_unified_views' else model end

where 1=1;
```

### Step 3: Setting up a new dbt project for Unified Digital

Due to a package conflict resulting from the last Web and Unified Digital releases requiring a different version of the `snowplow-utils` package, it's probably best to create a new dbt project for running the package.

**3.1 Packages.yml**

```yml
packages:

  - package: snowplow/snowplow_unified

    version: 0.4.1
```

**3.2 Project.yml**

In the project.yml file, disable the mobile events:

```yml
vars:

  snowplow_unified:

    snowplow__enable_mobile: false
```

If you changed any of the default variables in your web project, add them here as well. While 90% of the variables stayed the same, there are some notable changes, deletions, additions, and default modifications. Have a look below or at the detailed [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) page and adjust accordingly:

**Removed variables:**

- `snowplow__limit_page_views_to_session`

**Renamed variables:**

- `snowplow__page_view_stitching` to `snowplow__view_stitching`
- `snowplow__consent_cmp_visible` to `snowplow__cmp_visible_events` (Redshift only)
- `snowplow__consent_preferences` to `snowplow__consent_preferences_events` (Redshift only)
- `snowplow__cwv_context` to `snowplow__cwv_events` (Redshift only)

**Added variables:**

- `snowplow__conversion_stitching`
- `snowplow__conversion_passthroughs`
- `snowplow__enable_browser_context`
- `snowplow__enable_app_errors`
- `snowplow__enable_conversions`
- `snowplow__use_refr_if_mkt_null`
- `snowplow__browser_context`: com\_snowplowanalytics\_snowplow\_browser\_context\_1 (Redshift only)
- `snowplow__session_context`: com\_snowplowanalytics\_snowplow\_client\_session\_1 (Redshift only)
- `snowplow__mobile_context`: com\_snowplowanalytics\_snowplow\_mobile\_context\_1 (Redshift only)
- `snowplow__geolocation_context`: com\_snowplowanalytics\_snowplow\_geolocation\_context\_1 (Redshift only)
- `snowplow__application_context`: com\_snowplowanalytics\_mobile\_application\_1 (Redshift only)
- `snowplow__screen_context`: com\_snowplowanalytics\_mobile\_screen\_1 (Redshift only)
- `snowplow__application_error_events`: com\_snowplowanalytics\_snowplow\_application\_error\_1 (Redshift only)
- `snowplow__screen_view_events`: com\_snowplowanalytics\_mobile\_screen\_view\_1 (Redshift only)
- `snowplow__deep_link_context`: com\_snowplowanalytics\_mobile\_deep\_link\_1 (Redshift only)
- `snowplow__screen_summary_context`: com\_snowplowanalytics\_mobile\_screen\_summary\_1 (Redshift only)

**Variables with changed defaults:**

- `snowplow__session_identifiers`: `[{"schema" : "atomic", "field" : "domain_sessionid"}]`
- `snowplow__user_identifiers`: `[{"schema": "atomic", "field" : "domain_userid"}]` to none
- `snowplow__databricks_catalog`: `hive_metastore`

### Step 4: Verifying the new derived datasets

At this stage you could potentially run both packages simultaneously until you are happy to stop the old jobs that updated the Web package.

If you used the original derived tables as a source for your BI tool or for downstream data models, you can use the views below to replicate the original tables. This can help you maintain your existing reporting.

**SQL scripts to create views**

```sql
create view (your_schema)_derived.mock_snowplow_web_page_views as (

  select

    view_id as page_view_id,

    event_id,

    app_id,

    platform,

    user_id,

    user_identifier as domain_userid,

    device_identifier as original_domain_userid,

    stitched_user_id,

    network_userid,

    session_identifier as domain_sessionid,

    null as original_domain_sessionid,

    device_session_index as domain_sessionidx,

    view_in_session_index as page_view_in_session_index,

    views_in_session as page_views_in_session,

    dvce_created_tstamp,

    collector_tstamp,

    derived_tstamp,

    start_tstamp,

    end_tstamp,

    model_tstamp,

    engaged_time_in_s,

    absolute_time_in_s,

    horizontal_pixels_scrolled,

    vertical_pixels_scrolled,

    horizontal_percentage_scrolled,

    vertical_percentage_scrolled,

    doc_width,

    doc_height,

    content_group,

    page_title,

    page_url,

    page_urlscheme,

    page_urlhost,

    page_urlpath,

    page_urlquery,

    page_urlfragment,

    mkt_medium,

    mkt_source,

    mkt_term,

    mkt_content,

    mkt_campaign,

    mkt_clickid,

    mkt_network,

    default_channel_group,

    page_referrer,

    refr_urlscheme,

    refr_urlhost,

    refr_urlpath,

    refr_urlquery,

    refr_urlfragment,

    refr_medium,

    refr_source,

    refr_term,

    geo_country,

    geo_region,

    geo_region_name,

    geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_timezone,

    user_ipaddress,

    useragent,

    br_lang,

    br_viewwidth,

    br_viewheight,

    br_colordepth,

    br_renderengine,

    os_timezone,

    iab__category as category,

    iab__primary_impact as primary_impact,

    iab__reason as reason,

    iab__spider_or_robot as spider_or_robot,

    ua__useragent_family as useragent_family,

    ua__useragent_major as useragent_major,

    ua__useragent_minor as useragent_minor,

    ua__useragent_patch as useragent_patch,

    ua__useragent_version as useragent_version,

    ua__os_family as  os_family,

    ua__os_major as os_major,

    ua__os_minor as os_minor,

    ua__os_patch as os_patch,

    ua__os_patch_minor as os_patch_minor,

    ua__os_version as os_version,

    ua__device_family as device_family,

    yauaa__device_class as device_class,

    case

        when device_class = 'Desktop' then 'Desktop'

        when device_class = 'Phone' then 'Mobile'

        when device_class = 'Tablet' then 'Tablet'

        else 'Other'

    end as device_category,

    screen_resolution,

    yauaa__agent_class as agent_class,

    yauaa__agent_name as agent_name,

    yauaa__agent_name_version as agent_name_version,

    yauaa__agent_name_version_major as agent_name_version_major,

    yauaa__agent_version as agent_version,

    yauaa__agent_version_major as agent_version_major,

    yauaa__device_brand as device_brand,

    yauaa__device_name as device_name,

    yauaa__device_version as device_version,

    yauaa__layout_engine_class as layout_engine_class,

    yauaa__layout_engine_name as layout_engine_name,

    yauaa__layout_engine_name_version as layout_engine_name_version,

    yauaa__layout_engine_name_version_major as layout_engine_name_version_major,

    yauaa__layout_engine_version as layout_engine_version,

    yauaa__layout_engine_version_major as layout_engine_version_major,

    yauaa__operating_system_class as operating_system_class,

    yauaa__operating_system_name as operating_system_name,

    yauaa__operating_system_name_version as operating_system_name_version,

    yauaa__operating_system_version as operating_system_version,

  from (your_schema)_derived.snowplow_unified_views

);



create view (your_schema)_derived.mock_snowplow_web_sessions as (

  select

    app_id,

    platform,

    session_identifier as domain_sessionid,

    null as original_domain_sessionid,

    device_session_index as domain_sessionidx,

    start_tstamp,

    end_tstamp,

    model_tstamp,

    user_id,

    user_identifier as domain_userid,

    device_identifier as original_domain_userid,

    stitched_user_id,

    network_userid,

    views as page_views,

    engaged_time_in_s,

    total_events,

    is_engaged,

    absolute_time_in_s,

    first_page_title,

    first_page_url,

    first_page_urlscheme,

    first_page_urlhost,

    first_page_urlpath,

    first_page_urlquery,

    first_page_urlfragment,

    last_page_title,

    last_page_url,

    last_page_urlscheme,

    last_page_urlhost,

    last_page_urlpath,

    last_page_urlquery,

    last_page_urlfragment,

    page_referrer as referrer,

    refr_urlscheme,

    refr_urlhost,

    refr_urlpath,

    refr_urlquery,

    refr_urlfragment,

    refr_medium,

    refr_source,

    refr_term,

    mkt_medium,

    mkt_source,

    mkt_term,

    mkt_content,

    mkt_campaign,

    mkt_clickid,

    mkt_network,

    mkt_source_platform,

    default_channel_group,

    first_geo_country as geo_country,

    null as geo_region,

    first_geo_region_name as geo_region_name,

    first_geo_city as geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_timezone,

    first_geo_country_name as geo_country_name,

    first_geo_continent as geo_continent,

    last_geo_country,

    last_geo_region_name,

    last_geo_city,

    last_geo_country_name,

    last_geo_continent,

    user_ipaddress,

    useragent,

    br_renderengine,

    first_br_lang as br_lang,

    first_br_lang_name as br_lang_name,

    last_br_lang,

    last_br_lang_name,

    os_timezone,

    iab__category as category,

    iab__primary_impact as primary_impact,

    iab__reason as reason,

    iab__spider_or_robot as spider_or_robot,

    ua__useragent_family as useragent_family,

    ua__useragent_major as useragent_major,

    ua__useragent_minor as useragent_minor,

    ua__useragent_patch as useragent_patch,

    ua__useragent_version as useragent_version,

    null as  os_family,

    ua__os_major as os_major,

    ua__os_minor as os_minor,

    ua__os_patch as os_patch,

    ua__os_patch as os_patch_minor,

    os_version,

    ua__device_family as device_family,

    yauaa__device_class as device_class,

    device_category,

    screen_resolution,

    yauaa__agent_class as agent_class,

    yauaa__agent_name as agent_name,

    yauaa__agent_name_version as agent_name_version,

    yauaa__agent_name_version_major as agent_name_version_major,

    yauaa__agent_version as agent_version,

    yauaa__agent_version_major as agent_version_major,

    null as device_brand,

    yauaa__device_name as device_name,

    yauaa__device_version as device_version,

    yauaa__layout_engine_class as layout_engine_class,

    yauaa__layout_engine_name as layout_engine_name,

    yauaa__layout_engine_name_version as layout_engine_name_version,

    yauaa__layout_engine_name_version_major as layout_engine_name_version_major,

    yauaa__layout_engine_version as layout_engine_version,

    yauaa__layout_engine_version_major as layout_engine_version_major,

    yauaa__operating_system_class as operating_system_class,

    yauaa__operating_system_name as operating_system_name,

    yauaa__operating_system_name_version as operating_system_name_version,

    yauaa__operating_system_version as operating_system_version

  from (your_schema)_derived.snowplow_unified_sessions

);



create view (your_schema)_derived.mock_snowplow_web_users as (

  select

    user_id,

    user_identifier as domain_userid,

    null as original_domain_userid,

    network_userid,

    start_tstamp,

    end_tstamp,

    model_tstamp,

    views as pageviews,

    sessions,

    engaged_time_in_s,

    first_page_title,

    first_page_url,

    first_page_urlscheme,

    first_page_urlhost,

    first_page_urlpath,

    first_page_urlquery,

    first_page_urlfragment,

    first_geo_country,

    first_geo_country_name,

    first_geo_continent,

    first_geo_city,

    first_geo_region_name,

    first_br_lang,

    first_br_lang_name,

    last_geo_country,

    last_geo_country_name,

    last_geo_continent,

    last_geo_city,

    last_geo_region_name,

    last_br_lang,

    last_br_lang_name,

    last_page_title,

    last_page_url,

    last_page_urlscheme,

    last_page_urlhost,

    last_page_urlpath,

    last_page_urlquery,

    last_page_urlfragment,

    page_referrer as referrer,

    refr_urlscheme,

    refr_urlhost,

    refr_urlpath,

    refr_urlquery,

    refr_urlfragment,

    refr_medium,

    refr_source,

    refr_term,

    mkt_medium,

    mkt_source,

    mkt_term,

    mkt_content,

    mkt_campaign,

    mkt_clickid,

    mkt_network,

    mkt_source_platform,

    default_channel_group

    from (your_schema)_derived.snowplow_unified_users

);
```

---

# Configure custom aggregations for dbt models
> Details on custom aggregations in our packages
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/

While [passthrough fields](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) enable you to add fields from the event itself into derived tables, you may at times also wish to do aggregations on the identifier for that table, e.g. session identifier, to provide more information in your derived tables without having to make a full [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/). This is where Custom Aggregations come in; they allow you to calculate aggregations directly as part of the derived table logic, without having to build your own models.

## Availability

| Package | Minimum Required Version |
| ------- | ------------------------ |
| Unified | 0.3.0                    |

## Usage

To use a the custom aggregation, you need to set the relevant variable in your root `dbt_project.yml` file; e.g `snowplow__session_aggregations` (see your package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for a full list of aggregation variables).

These variables are lists of objects that define your aggregations; they have keys of the `type` of aggregations you wish to do, the `field` to aggregate over, and the `alias` to give your aggregation in the derived table.

The `field` can be any valid column sql, including `case when` statements or making use of other sql functions. For example, to count the number of `purchase` events over a session you would have:

```yml
vars:

  snowplow_unified:

    snowplow__view_aggregations: [{'type': 'sum', 'field': "case when event_name = 'purchase' then 1 else 0 end", 'alias': 'num_purchase_events'}]
```

The aggregation always runs on the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table so all events (with a session identifier) and columns are available for you to use.

Note that how to extract and use field from your entity or self-describing event columns will depend on your warehouse (see our [querying guide](/docs/destinations/warehouses-lakes/querying-data/#entities) for more information), and you are unable to use dbt macros in this variable.

### Supported Aggregations

Currently we support the following aggregations:

| Aggregation    | `type` value |
| -------------- | ------------ |
| Count          | `count`      |
| Sum            | `sum`        |
| Minimum        | `min`        |
| Maximum        | `max`        |
| Average        | `avg`        |
| Count Distinct | `countd`     |

## Usage Notes

> **Note:** It is unlikely, although not impossible, that when using the SQL approach you may need to provide a table alias to avoid ambiguous references, in this case please see the model sql file for the specific alias used for the `snowplow_unified_base_events_this_run` table in each case.

---

# Configure custom sessionization
> Define custom session and user identifiers in Snowplow dbt packages using SQL expressions and custom logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/

> **Tip:** As this changes the core logic of the package, you should make sure you have a good understanding of how the [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) works (including things such as quarantining sessions), and a good certainty around the tracking of any custom fields you plan to use.

Our packages come with a very robust [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) that makes sure when a session has a new event in the latest run, that _all_ events within that session are processed as part of that run to ensure all aggregations are correct and valid. There are some exceptions, such as quarantined sessions to avoid large table scans, but in general this approach ensures that you can be confident that in each run you have the full scope of a given session to process in our models, as well as any [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) you build to take advantage of this feature.

Traditionally a session in this sense has been a _true_ session i.e. it has been the `domain_sessionid` and `device_sessionid` value that has identified which events belong to the same session, however in our packages it is possible to overwrite the default field as part of the package logic and use your own session identifier, to ensure all events within a "session" are processed in the same run, whatever a "session" in this case means to you.

> **Info:** Throughout this page we refer to entity columns only by their major version (e.g. `com_mycompany_session_identifier_1`). If you use BigQuery, you will need to adjust these for the full version number (or use the `_*` wildcard to denote all minor versions), and may need to manage the combination of different versions yourself, details of how can be found below.

> **Tip:** Remember that any events with a null "session" identifier will be excluded from the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table and any processing of the package, make sure your identifier has a value for all events you want to process!

## Customizing session identifiers

To customize your session identifiers, you can make use of the `snowplow__session_identifiers` variable. This variable allows you to provide a list of identifiers that dbt will then try to use to create an identifying field for each session (by coalescing them in order), which will always be saved under the `session_identifier` column in your tables.

### Using additional atomic fields

For web data by default, your identifier will be the `domain_sessionid` field which is found in the atomic events table (this will vary by package). If you wanted to instead use a different field, say the `domain_userid` field that can also be found in the flat fields of the atomic events table, you could define your `snowplow__session_identifiers` as follows:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'atomic', 'field': 'domain_userid'}]

    ...

...
```

> **Tip:** `schema` here refers to the schemata the data was created from i.e. an atomic field, a self describing event, or an entity value. This has no relation to the database schema the table is in.

If you wanted to include multiple identifiers, in case the first one is sometimes null, then you could define the `snowplow__session_identifiers` as follows:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'atomic', 'field': 'domain_userid'},

                                    {'schema': 'atomic', 'field': 'domain_sessionid'}]

    ...

...
```

This would then compile into the following SQL code:

```sql
SELECT

    ...

    COALESCE(e.domain_userid, e.domain_sessionid, NULL) as session_identifier,

...

FROM events e
```

> **Info:** The order in which you provide your identifiers is the order of precedence they will take in the `COALESCE` statement. In other words, in the example above the value of `domain_sessionid` will **only** be used when `domain_userid` is `NULL`. If both are `NULL`, the `session_identifier` will be `NULL` for that event, and that event will not be processed with this package.

### Using custom entities

If you wanted to instead use session identifiers that come from a custom entity, you can do that as well. Let's assume you've created a session identifier entity that you attach to your events, and it's called `com_mycompany_session_identifier_1`. Let's also say you're interested in the `session_id` field of this entity as the identifier for each session. To make sure that your dbt models extract and use this field as the value for `session_identifier`, you need to define the `snowplow__session_identifiers` as follows:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_1', 'field': 'session_id', 'prefix': 'si'}]

    ...

...
```

> **Tip:** Make sure you include a `prefix` value if you are running on **Postgres or Redshift**, as this ensures that you don't have duplicate column names somewhere in your SQL select statement. It is not required for the other warehouses, but is recommended.

Similar to before, if you want to combine multiple identifiers in different (or the same) entities, you can do so by defining your `snowplow__session_identifiers` as shown below. First, however, let's assume there's another entity called `com_mycompany_logged_session_id_1` which has both a `logged_in_id` and `session_identifier` field in it. To include all of these:

```yml
vars:

    ...

    snowplow__session_identifiers: [

            {'schema': 'com_mycompany_logged_session_id_1', 'field': 'logged_in_id', 'prefix': 'lsi'},

            {'schema': 'com_mycompany_logged_session_id_1', 'field': 'session_identifier', 'prefix': 'lsi'},

            {'schema': 'com_mycompany_session_identifier_1', 'field': 'session_id', 'prefix': 'si'}]

    ...

...
```

> **Tip:** Make sure if you include multiple fields from the same entity that you give them the same prefix.

This will then render into the following SQL:

**Databricks & Snowflake:**

```sql
SELECT

    ...

    COALESCE(

        com_mycompany_logged_session_id_1[0].logged_in_id,

        com_mycompany_logged_session_id_1[0].session_identifier,

        com_mycompany_session_identifier_1[0].session_id,

        NULL

        ) as session_identifier,

...
```

**Bigquery:**

```sql
SELECT

    ...

    COALESCE(

        com_mycompany_logged_session_id_1_0_0[safe_offset(0)].logged_in_id,

        com_mycompany_logged_session_id_1_0_0[safe_offset(0)].session_identifier,

        com_mycompany_session_identifier_1_0_0[safe_offset(0)].session_id,

        NULL

        ) as session_identifier,

...
```

**Redshift & Postgres:**

```sql
SELECT

    ...

    COALESCE(

        lsi_logged_in_id,

        lsi_session_identifier,

        si_session_id,

        NULL

        ) as session_identifier,

...
```

Note that we will manage joining any self describing event or entity tables for you as part of the package logic.

***

Again the order of precedence being decided by the order of your list of identifiers.

### Schema evolution of custom entities

> **Tip:** You can use this approach for schema evolution in BigQuery for any changes, in other warehouses you would only need to do this for major version changes.

This can be extended into entities where schema evolution takes place. Suppose your session identifying entity has previously always been `com_mycompany_session_identifier_1`, where you extract the `session_id` field as the identifier. Then suppose you introduce some breaking changes that cause you to now use the `com_mycompany_session_identifier_2` entity, where you extract `new_session_id` as the identifying field. In order to ensure you can track both as your session identifier, you would define your `snowplow__session_identifiers` as follows:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_2', 'field': 'new_session_id', 'prefix': 'si_t'},

                                    {'schema': 'com_mycompany_session_identifier_1', 'field': 'session_id', 'prefix': 'si_o'}]

    ...

...
```

This setup implies that the `new_session_id` from `com_mycompany_session_identifier_2` should have precedence over the `session_id` field from `com_mycompany_session_identifier_1`, meaning that if both are filled for a particular event, the `new_session_id` value would be the one present in the `session_identifier` field in your dbt tables. If you'd prefer to have the precedence swapped, you can swap the ordering in the variable in your `dbt_project.yml`.

**BigQuery Version Changes**

Handling minor version bumps of your schemas where this isn't automatically handled by the loader, such as in BigQuery, works in the exact same way. A small example of bumping `com_mycompany_session_identifier_1_0_0` to `com_mycompany_session_identifier_1_1_0` but leaving the `session_id` field as the key identifier would look as follows:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_1_1_0', 'field': 'session_id', 'prefix': 'si_t'},

                                    {'schema': 'com_mycompany_session_identifier_1_0_0', 'field': 'session_id', 'prefix': 'si_o'}]

    ...

...
```

Alternatively you can use a `_*` in the schema name to get the field from all columns of the same major version:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_1_*1_0*', 'field': 'session_id', 'prefix': 'si'}]

    ...

...
```

### Combining atomic fields and custom entities

Combining atomic fields and custom entities is straightforward if you're comfortable with what we described above. Let's say you want to combine the `logged_in_id` field from the `com_mycompany_logged_session_id_1` entity together with the standard `domain_sessionid` field in the `events` table. To achieve that, you would include the following in your `dbt_project.yml`:

```yml
vars:

    ...

    snowplow__session_identifiers: [{'schema': 'com_mycompany_logged_session_id_1', 'field': 'logged_in_id', 'prefix': 'lsi'},

                                    {'schema': 'atomic', 'field': 'domain_sessionid', 'prefix': 'e'}]

    ...

...
```

### Adding your own custom session logic

If there are session identifiers that are more complicated to utilize, then you can also provide your own session logic that will be used instead of the logic explained in the preceding sections. As an example, if you would want to concat two fields to create a session identifier, or instead apply a SQL function to a field to then use as a session identifier, that is completely possible using the `snowplow__session_sql` variable.

> **Info:** Defining the `snowplow__session_sql` variable will ensure that the package takes it's value as the `session_identifier` **over** anything you may have defined with the `snowplow__session_identifiers` variable.

> **Note:** For Redshift/Postgres, if you want to leverage custom entities for your custom session logic, you will need to include them in the `snowplow__session_identifiers` variable in the same way as in previous sections to ensure they are correctly joined to the table.

#### Concatenating multiple fields to create a session identifier

To start, suppose you want to combine the atomic `domain_sessionid` and `domain_userid` fields to create a session identifier. It's simple to do that by defining the following variable in your `dbt_project.yml`:

```yml
vars:

   ...

   snowplow__session_sql: "e.domain_userid || '_' || e.domain_sessionid"

   ...

...
```

This would be parsed into the following SQL:

```sql
SELECT

    ...

    e.domain_userid || '_' || e.domain_sessionid as session_identifier,

...
```

#### Applying a SQL function to a field to use as a session identifier

Instead, suppose you want to take the `DATE` value of your `derived_tstamp` as your session identifier. It's also simple to do that by defining the following variable in your `dbt_project.yml`:

```yml
vars:

   ...

   snowplow__session_sql: "DATE(e.derived_tstamp)"

   ...
```

This would be parsed into the following SQL:

```sql
SELECT

    ...

    DATE(e.derived_tstamp) as session_identifier,

...
```

---

# Users and identity stitching
> Details on mapping between `domain_userid` and `user_id` in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/

**Identity stitching is the process of taking various user identifiers and combining them into a single user identifier, to better identify and track users throughout their journey on your site/app.**

Stitching users together is not an easy task: depending on the typical user journey the complexity can range from having individually identified (logged in) users, thus not having to do any extra modeling to never identified users mainly using the same common public device (e.g. school or library) where it is technically impossible to do any user stitching. As stitching is a reiterative process as it constantly needs to be updated after each incremental run for a desirably large range of data, compute power and extra expenses as well as time constraints may limit and dictate the best course of action.

## Session stitching

For the out-of-the-box user stitching we opt for the sweet spot method: applying a logic that the majority of our users will benefit from while not introducing compute-heavy calculations. We do this in our core dbt package, the `Snowplow Unified Digital` package that produces a `users` table.

We provide user stitching on one layer: between the primary `user_identifier` field and a custom defined (`var('snowplow__user_stitching_id')`) id field. The package considers the `user_identifier` field as the primary key for the users table and this would typically be the most "reliable" cookie based field, hence it is defaulted to `domain_userid` when processing web events and the session\_context entity based user\_id for mobile events. The custom defined id to be used for stitching is recommended to be the `atomic.user_id` field that is used for tracking logged in users. This reliable user identifier field is the common denominator for applying the so called session-stitching in our packages.

This works by having an `User Mapping` table that collects and incrementally updates the latest official logged in `user_id` field for all user\_identifiers processed in the current run (taken from the event\_this\_run table). This mapping is then applied to the derived tables (e.g views table if `snowplow__view_stitching` is enabled and both the sessions and users table if `snowplow__session_stitching` is enabled). The update is carried out by a post-hook (defined in the config of each derived incremental table), which updates the `stitched_user_id` column with the latest mapping. If no mapping is present, the default value for `stitched_user_id` is the user identifier itself. This process is known as session stitching, and effectively allows you to attribute logged-in and non-logged-in sessions back to a single user.

## Cross platform stitching

The `snowplow_unified` package means that all the user data, from both web and mobile, is modeled in one place. This makes it easy to effectively perform cross-platform stitching, which means that as soon as a users identify themselves by logging in as the same user on separate platforms, all the user data will be found within one package making it convenient for performing further analysis. We encourage everyone to take the base stitching logic provided by the package further by applying a custom aggregation downstream layer that takes the first/last fields per stitched\_user\_id from the users table as well a applying additional stitching based on custom user\_mapping table(s) depending on need.

> **Info:** In order for the user\_mapping table to successfully update the stitched\_user\_id field in the users table, it needs to stay unique on session\_identifier which means that if for some reason there are multiple user\_ids per user\_identifier, we take the latest one. For a standard use case when the user\_identifier is a cookie based domain\_userid and the user\_stitching\_id is the logged in atomic.user\_id field, we encourage everyone to [remove cookies](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) to force creating a new session upon the user log out action to avoid a scenario when multiple users log in and out quickly after one another on the same device (e.g. school, library). For tracking advice, have a look at [this](/docs/events/identifiers/#reset-generated-identifiers-on-web) documentation.

![Session stitching in the Unified Digital Model](/assets/images/session_stitching_light_unified.drawio-e4c7cb45d21c40e619445fcb19a58f04.png)

If required, this update operation can be disabled by setting in your `dbt_project.yml` file (selecting one of web/mobile, or both, as appropriate):

```yml
vars:

  snowplow_<package>:

    snowplow__session_stitching: false
```

> **Tip:** Consider processing costs before enabling `snowplow__view_stitching` to `true`. It may be enough to apply this with less frequency than on sessions to keep costs down, by only enabling this at runtime on only some of the runs.

### Custom solutions

#### **Customizing user identifiers**

Customizing user identifiers works in the exact same way as [customizing session identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/#customizing-session-identifiers), please refer to that link to understand the breakdown of how to set this up, although you need to make use of the `snowplow__user_identifiers` variable instead of the `snowplow__session_identifiers`, and `snowplow__user_sql` in place of `snowplow__session_sql`.

```yml
vars:

  snowplow_unified:

    # This is an example of user identifiers for BigQuery

    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2_*", "field" : "user_id"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1_*", "field" : "user_id"}]

    # For Databricks

    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2", "field" : "user_id"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1", "field" : "user_id"}]

    # For Redshift/Postgres

    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2", "field" : "user_id", "prefix" : "ui_t", "alias": "uidt"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1", "field" : "user_id", "prefix": "ui_o", "alias": "uido"}]

    # For Snowflake

    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2", "field" : "userId"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1", "field" : "userId"}]
```

If you need a specific way to refer to a custom user you can also use the `snowplow_user_sql` variable, which will override any default or overwrites on `snowplow__user_identifiers`.

Please note, however, that the `snowplow__user_identifiers` variable is not designed to handle user stitching the way you might expect it. Currently, there is a limitation that if any of the identifiers in the list do not persist throughout the session, there is no guarantee that the highest-prestige identifier (first in the list) will take precedence.

This is particularly relevant for those adding user\_id as the first element in their list, as it may be NULL for some events within a session. For that it is best to rely on the package based user stitching logic, leave user\_id as the `snowplow__user_stitching_id` and rely on the `stitched_user_id` field produced by the package in each derived tables as the main identifier field. This way even if the user logged in for some period of the time within a session the user\_id field will be found and prioritized over the rest.

Alternatively, to ensure a deterministic user selection, you could rely on the `snowplow_user_sql` variable instead to something like this:

```sql
COALESCE(

    MAX(user_id),

    MAX(domain_userid),

    MAX(network_userid)

) OVER (PARTITION BY session_identifier)
```

This ensures that `MAX(user_id)` is computed separately before applying COALESCE() at the session level, preventing another identifier from accidentally taking precedence.

#### **Handling Anonymized Users**

In case of applying Client-side anonymization with session tracking, the `userId` property of the `contexts_com_snowplowanalytics_snowplow_client_session_1` equates to a null UUID which will appear as `00000000-0000-0000-0000-000000000000` in the database. It may be convenient to make this field an actual NULL field to make it easier to exclude them from modeling (e.g the user mapping table of the Unified Package excludes null values). This can be made possible with the use of the `snowplow__user_sql` variable, however, this means that the extraction from the relevant context/sde field needs to be handled manually.

Example implementation (Snowflake):

```yml
vars:

  snowplow_unified:

    snowplow__user_sql: 'coalesce(case when contexts_com_snowplowanalytics_snowplow_client_session_1[0]:userId::varchar(36) == "00000000-0000-0000-0000-000000000000" then null else contexts_com_snowplowanalytics_snowplow_client_session_1[0]:userId::varchar(36) end, domain_userid)'
```

> **Info:** Defining the `snowplow__user_sql` variable will ensure that the package takes its value as the `user_identifier` **over** anything you may have defined with the `snowplow__user_identifiers` variable.

User mapping is typically not a one-size-fits-all exercise. Depending on your tracking implementation, business needs and desired level of sophistication you may want to write bespoke logic. Please refer to this [blog post](https://snowplow.io/blog/developing-a-single-customer-view-with-snowplow/) for ideas. The unified package offer the ability to change what field is used as your stitched user id, so instead of `user_id` you can use any field you wish (note that it will still be called `user_id` in your mapping table), and by taking advantage of the [custom sessionization and users](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) you can also change the field used as the user\_identifier (unified model).

### Overview

The below diagram shows a potential flow your user may take across multiple devices. It does not matter if they are web or mobile events as Unified will correctly process and stitch both. As the user progresses through the (simplified) sessions table tracks their sessions, user identifier, user ID, and stitched user id. Once a user ID is identifier for specific user identifier it is backdated in the stitched column for all sessions with that identifier. Note that this is not possible _until_ the user logs in during a session.

![Overview of stitching scenarios](/assets/images/stitching_scenarios.drawio-4e90f4a1e53df103697695a6ca3f0d07.png)

---

# Snowplow dbt package features
> Top level page for features contained in our packages
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/

Our packages contain a wide range of functionality and features you can use to configure the package to suit your needs. Details of these features can be found in the following pages.

---

# Model entities and self-describing events
> Model entities and self-describing events in Snowplow dbt packages with out-of-the-box support and custom modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/

Our dbt packages provide out-of-the-box modeling for many [entities](/docs/fundamentals/entities/) that are relevant to each package. Fields from these contexts are added to the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table for use within downstream models. For example our [unified package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) will add all fields from the [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) entity by setting the `snowplow__enable_yauaa` to true.

Relevant [self-describing events](/docs/fundamentals/events/#self-describing-events) fields are extracted in the same way as needed per-package, and are often dependent on enabling optional modules.

> **Tip:** In some cases, these fields may exist but be null when the entity or self-describing event is not enabled in the project yaml to enable more consistent structure to the events this run table.

## Custom Entities & Events

For warehouses other than Redshift, all columns in your atomic events table are available in the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table, so you can use them as needed in any other models.

For Redshift we provide the `snowplow__entities_or_sdes` in our packages that have an events this run table. This variable takes the name of your entity or self-describing event table, a prefix, an alias, and if it is a single entity.

> **Warning:** Only entities that are "single valued" should be added directly to the event this run table i.e. the list should only contain one instance of the entity. A multi-valued entity would cause duplicates in the events this run table and lead to incorrect data in all downstream models. See below for how to use multi-valued entities.

This variable is a list of objects which have the following structure:

| Key             | Description                                                                                                                                                                                                  | Example                          |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------- |
| `schema`        | The schema denotes the name of the entity or SDE that you would like to join, which should also be the name of the table that is in your warehouse.                                                          | `contexts_com_mycompany_click_1` |
| `prefix`        | The prefix that each field in the context will receive. E.g with a prefix of `my_click` and a field name of `id`, this field will be accessible in the events this run table under the `my_click_id` column. | `my_click`                       |
| `alias`         | The alias that is used for the context table join, for reference in your custom SQL queries.                                                                                                                 | `mc`                             |
| `single_entity` | A boolean to say whether this is a single entity or whether there can be multiple for each event. Should always be set to `true`.                                                                            | `true`                           |

An example with two custom contexts would be:

```yaml
...

vars:

  snowplow_<package_name>:

    snowplow__entities_or_sdes: [

        {'schema': 'contexts_com_mycompany_click_1', 'prefix': 'click', 'alias': 'mc', 'single_entity': true},

        {'schema': 'contexts_com_mycompany_submit_1', 'prefix': 'submit', 'alias': 'sub', 'single_entity': true}

        ]

...
```

### Multi-valued Entities

In the case of a mutli-valued entity, you must make use of custom models and the `snowplow_utils.get_sde_or_context` to manage de-duplication. See the [using multi-valued entities](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/) custom model example page for more information.

---

# Passthrough fields for derived tables
> Details for how to pass additional fields through to the derived tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/

Passthrough fields are the term used to define any field from your [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) type table that you want available _as is_ in a derived table, i.e. these fields are _passed through_ the processing to be available in your derived tables. There are many use cases for this, but the most common is the need to add a field from a custom context to the e.g. the views table to use as an additional dimension in analysis. Previously you would have to write a custom model to join this field on, or join the original view event at the time of the analysis, but with passthrough fields this is now far easier (and cheaper) to achieve.

## Availability

| Package      | Minimum Required Version |
| ------------ | ------------------------ |
| Unified      | 0.3.0                    |
| Media Player | 0.7.0                    |
| Ecommerce    | 0.6.0                    |

## Usage

To enable the passthrough fields, you need to set the relevant variable in your root `dbt_project.yml` file. E.g. `snowplow__view_passthroughs` (see your package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for a full list of passthrough variables).

> **Info:** Note that in some cases you may be able to specify multiple variables for the same table for the first and last of a given record (e.g. first and last session for a user).

To set these variables, you need to define field references in the format of a list of `field names` (in case it is one of the event table columns that can be used as is without the need to alias it) or `dictionaries` specifying the `sql` which defines the sql creating a new field, as well as the `alias` to give a name to the custom field name. The list of such field references to be used as passthrough fields will get compiled and used in a select clause, therefore any valid sql within that context is applicable.

### 1. Passing through columns as is

In case of out-of-the-box event table columns, you can simply reference the name of the field as it appears as a column within your events table. E.g.:

```yml


snowplow__view_passthroughs: ['page_url', 'page_title']
```

For those warehouses where the self-describing events or custom entities are also available in the events table you can opt for unnesting the column in a downstream table only, passing through the unnested field as is. E.g.:

```yml
snowplow__view_passthroughs: ['contexts_my_entity_1']
```

### 2. SQL-defined Columns

You can also use any valid sql supported by your warehouse that can be used in a select clause. This time you need to use the dictionary format, specifying the relevant sql and the field alias. E.g.:

```yml
snowplow__view_passthroughs: [

  \{'sql': 'COALESCE(page_url, refr_url)', 'alias': 'final_url'}

]
```

### 3. Extracting fields from a self-describing event or custom context entity

A more useful case for the SQL block is to extract a specific field from an entity or sde. However, please note that how to extract a field from your self describing event / entity column will depend on your warehouse:

**Redshift, Postgres:**

**Step 1. Making fields available in the events table**

For Redshift and Postgres users, entities and self describing events are not part of the standard events table. Instead, each type of entity/sde is in its own table. The table name and the fields in the table will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

In order for you to use fields from there through passthrough fields, you would need to first make sure that those fields are part of the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table. Any custom entities or self-describing events can be added to this table (which get de-duped by taking the earliest `collector_tstamp` record) by using the `snowplow__entities_or_sdes` variable in our package. See [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for more information and examples.

> **Note:** It is unlikely, although not impossible, that when using the SQL approach you may need to provide a table alias to avoid ambiguous references, in this case please see the model sql file for the specific alias used for the `snowplow_unified_base_events_this_run` table in each case.

```yml
snowplow__entities_or_sdes: [

      \{'schema': 'custom_table_name', 'prefix': 'my_entity_1', 'alias': 'entity_1', 'single_entity': true},

]
```

**Step 2. Referencing the fields as passthrough fields**

Once you add those fields into your events\_this\_run table through the `snowplow__entities_or_sdes` variable, you can simply refer to them as passthrough fields, coalesce different versions etc.:

```yml
snowplow__view_passthroughs: [

  'my_entity_1_field_x',

  'my_entity_1_field_y',

  'my_entity_1_field_z',

  \{'sql': 'COALESCE(my_entity_1_field_x, my_entity_1_field_y)', 'alias': 'entity_field_combined'}

]
```

**BigQuery:**

**Self-describing Events**

```yml
snowplow__view_passthroughs: [

  # Extract specific fields from self-describing events

  {'sql': 'unstruct_event_my_event_1_0_0.field_name', 'alias': 'my_event_field'},

  # Custom sql example (joining two extracted fields together)

  {'sql': 'unstruct_event_my_event_1_0_0.field1 || unstruct_event_my_event_1_0_0.field2', 'alias': 'my_event_fields'}

]
```

**Context Entities**

```yml
snowplow__view_passthroughs: [

  # Extract specific fields from a custom context field

  {'sql': 'contexts_my_entity_1_0_0[SAFE_OFFSET(0)].field_name', 'alias': 'entity_field'},

  # Custom sql example (coalescing different major versions of the same field)

  \{'sql': 'COALESCE(contexts_entity_1[SAFE_OFFSET(0)].field, contexts_entity_2[SAFE_OFFSET(0)].field)', 'alias': 'entity_field_combined'}

]
```

**Snowflake:**

**Self-describing Events**

```yml
snowplow__view_passthroughs: [

  # Extract specific fields from self-describing events

  {'sql': 'unstruct_event_my_event_1:fieldName::varchar', 'alias': 'event_field'},

  # Custom sql example (joining two extracted fields together)

  {'sql': 'unstruct_event_my_event_1:field1::varchar || unstruct_event_my_event_1:field2::varchar', 'alias': 'combined_fields'}

]
```

**Context Entities**

```yml
snowplow__view_passthroughs: [

  # Extract specific fields from a custom context field

  {'sql': 'contexts_my_entity_1[0]:fieldName::varchar', 'alias': 'entity_field'},

  # Custom sql example (coalescing different major versions of the same field)

  \{'sql': 'COALESCE(contexts_my_entity_1[0]:field::varchar, contexts_my_entity_2[0]:field::varchar)', 'alias': 'entity_field_combined'}

]
```

**Databricks, Spark SQL:**

**Self-describing Events**

```yml
snowplow__view_passthroughs: [

  # Extract specific fields from self-describing events

  {'sql': 'unstruct_event_my_event_1.field_name', 'alias': 'event_field'},

  # Custom sql example (joining two extracted fields together)

  {'sql': 'unstruct_event_my_event_1.field1 || unstruct_event_my_event_1.field2', 'alias': 'combined_fields'}

]
```

**Context Entities**

```yml
snowplow__view_passthroughs: [

  # Extract specific fields from a custom context field

  {'sql': 'contexts_my_entity_1[0].field_name', 'alias': 'entity_field'},

  # Custom sql example (coalescing different major versions of the same field)

  \{'sql': 'COALESCE(contexts_my_entity_1[0].field, contexts_my_entity_2[0].field)', 'alias': 'entity_field_combined'}

]
```

***

> **Tip:** When using passthrough fields with entities or self-describing events, always consider:
>
> 1. Field type casting may be required (especially in Snowflake)
> 2. Array indexes start at 0 for the first occurrence
> 3. Use COALESCE when dealing with multiple versions of the same field
> 4. Please note you are unable to use dbt macros in this variable, and also not able to use lateral flatten / explode type extractions.

## Usage Notes

The event that the passthrough fields are taken from depends on the derived table you are adding them to. For example, in the `snowplow_unified_views` table, the `page_view` event is used, not other events, such as `page_ping` or any other custom events. Therefore, if you are adding a passthrough field to the `snowplow_unified_views` table, you must access a property or custom entity that is attached to a `page_view` event.

A general rule of thumb is that the field comes from the _first_ event of that type at that table level. If you are unsure, you can always check the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for the variable description or the model sql.

As mentioned, the fields are passed through _as is_, which means it is not currently possible aggregate the value of a field across a page view or session as we do some other fields in the table. In the case where this is required you should use the [custom aggregations](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/) if that is supported by the package, or build a [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/).

> **Note:** In certain cases, such as the users table in Unified, it may be required to first set passthrough fields on an upstream table (the sessions table in that case) to make sure it is available for selection. The best way to identify this is to look at the DAG in the dbt docs for the package you are using.

---

# Table grants for dbt packages
> Details for granted access to our tables to other users/roles
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/table-grants/

> **Tip:** This functionality requires dispatching to our macros over dbt core, see how to do this on the [dispatch setup](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/dispatch/) page.

## Availability

> **Tip:** Any package can make use of the table grants feature provided by the `snowplow__grant_select_to` variable if you are using at least version 0.16.2 of snowplow-utils, even if the variable is not listed in the configuration. Granting usage on schemas however requires specific package versions.

| Package         | Minimum Required Version |
| --------------- | ------------------------ |
| Unified Digital | 0.3.0                    |
| Ecommerce       | 0.8.1                    |
| Media Player    | 0.7.2                    |
| Normalize       | 0.3.5                    |
| Attribution     | 0.2.0                    |

Note that this feature is not supported for BigQuery due to the different approach to permissions they take via IAM roles.

## Usage

### Granting select on tables

To grant `select` on all tables created within our package, you can provide a list of the users/roles in the `snowplow__grant_select_to` variable e.g.

```yaml
vars:

  snowplow_<package_name>:

    snowplow__grant_select_to: ['myuser1', 'myuser2']
```

Note that these user/role names are case sensitive. Databricks Principals are also supported. If the user does not exist an error will occur when the grant tries to run. This feature is compatible with the built-in [dbt grants](https://docs.getdbt.com/reference/resource-configs/grants) functionality and we will grant to a combination of the two.

> **Tip:** It is important to scope this to the relevant package, if you set this variable at the top level of your project then all models will have this grant applied.

> **Note:** Note this will overwrite any existing grants applied to the table manually in the warehouse.

### Granting usage on schemas

In the case of some warehouses, users also need `usage` permissions on the schema a table is in to be able to access the data. We provide this functionality via a post-hook that grants usage on **any** schema interacted with during the `dbt run` to the users listed in `snowplow__grant_select_to`.

> **Note:** Due to limitations with dbt, we are not able to scope this only to schemas interacted with by our package. This means **ALL** schemas in the run will have `usage` granted to these users.

This functionality will only trigger if `snowplow__grant_select_to` is not empty, and you can disable this by setting `snowplow__grant_schema_usage` to false.

---

# Snowplow dbt packages deduplicate events
> How we deal with duplicate events in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/

Duplicates in your data can happen, because of Snowplow's _At Least Once_ delivery approach. However, it's important for analysis and reporting that these duplicates are removed when doing any kind of data modeling. As part of our packages, we make sure to remove duplicate events at the very start to ensure any downstream data is correct.

All our packages perform de-duplication on both `event_id`'s as part of the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run), and then ensure distinct counts and records are used for any higher level tables in the packages e.g. page views. The de-duplication method for Redshift & Postgres is different to BigQuery, Snowflake, & Databricks due to their shredded table design.

## BigQuery, Snowflake, & Databricks

Any duplicate `event_id`s are removed by taking the earliest `collector_tstamp` record. In the case of multiple rows with this timestamp, we take one at random (as they will all be the same). The same methodology is applied to `page/screen_view_id`s, however we order by `derived_tstamp`.

There should be no need to further de-duplicate the base [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table or any contexts for each row.

## Redshift & Postgres

### Single-valued entities

When events are expected to only have at most a single entity in the attached context, any duplicate `event_id`s are removed from the atomic events table, and any context/self-describing event tables, by taking the earliest `collector_tstamp` record. In the case of multiple rows with this timestamp, we take one at random (as they will all be the same).

Any custom entities or self-describing events are able to be added to the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table, and are de-duped on by taking the earliest `collector_tstamp` record, by using the `snowplow__entities_or_sdes` variable in our package. See [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for more information and examples.

### Multi-valued entities

In the case where it may be possible for a entity to contain multiple instances (e.g. products returned in a search result) a more complex approach is taken to ensure that all of the instances in the attached entity are still in the table before the join, even in the case where some of these entities may be identical.

When we de-duplicate the events this run table we keep the number of duplicates there were. In the de-duplication of the entity table we generate a row number per unique combination of **all** fields in the record. A join is then made on `root_id` and `root_tstamp` as before, but with an **additional** clause that the row number is a multiple of the number of duplicates to support the 1-to-many join. This ensures all duplicates are removed while retaining all original instances of the entity. Because this leads to multiple rows per event, this is only ever done in downstream models in our packages.

If you need to make use of a multi-valued entity in a custom model, see the [using multi-valued entities](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/) custom model example page.

---

# Dispatch setup for dbt packages
> Details on how to setup the dispatch order for macros
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/dispatch/

Some of the functionality of our packages overwrites the default functionality of dbt core. To enable the use of our version over dbt, you must add the following must be added to your `dbt_project.yml` file once.

```yml
dispatch:

  - macro_namespace: dbt

    search_order: ['snowplow_utils', 'dbt']
```

This will ensure that, when it exists, our version of a macro is used over the dbt-core version.

---

# Full calculation for run timestamps
> A detailed deep dive into how we calculate which events to process in a run.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/full-run-calculation/

> **Tip:** The following information is meant for educational purposes only, there should never be a need to alter this logic as the package manages it all for you.

## Timestamps used in our packages

When calculating the timestamps required for our [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) by default we use `collector_tstamp`; historically this was the field the events table was most likely to be partitioned on. Nowadays `load_tstamp` is increasingly common, and users may have changed the partition field of their table. To accommodate this and make use of more efficient filtering you can alter the field we use to identify new events using the `snowplow__session_timestamp` variable. Note that where possible this should match your partition key, and the choice will impact the calculation when looking for which events to process.

## Calculation for run timestamps

The details provided above cover how we calculate the first step of the date range to process on a run, based on the state of models in the package and any new data. This is not the full story as in a given run the date range of specific events that are available in the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table will differ from this. What follows is the full details of how this is calculated and processed.

### Step 1: Calculate Base New Event Limits

The first step is to identify the model limits for the run. This is accomplished by the [`get_incremental_manifest_status` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/incremental_hooks/get_incremental_manifest_status.sql) to find the min and max last success in the manifest, and then uses the [`get_run_limits` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/incremental_hooks/get_run_limits.sql) to identify which state the package is in and calculate the timestamp range for this run, as detailed above.

This range is then printed to the console and stored in the `snowplow_<package_name>_base_new_event_limits` table for later use.

### Step 2: Update sessions lifecycle manifest

Next, we need to now process the sessions with new events into the [sessions lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#sessions-lifecycle-manifest) table. Here we apply many filters to the events table before calculating the [identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) and minimum and maximum `snowplow__session_timestamp` for each session.

The filters that are applied are:

#### Filter to events within the limits

To minimize the table scan and ensure a consistent processing of data, we filter to events with a timestamp bound by the limits defined in step 1.

```jinja2
and {{ session_timestamp }} >= {{ lower_limit }}

and {{ session_timestamp }} <= {{ upper_limit }}
```

If `snowplow__derived_tstamp_partitioned` is set to true, and it is running in BigQuery, this will also apply the filter to the `derived_tstamp` field as well.

#### Filter late arriving data

To avoid scanning too far back in the events table for a session, and to avoid recalculating sessions far in the past, we filter out late arriving data based on the device created and send timestamp. How late these events can be sent is defined by the `snowplow__days_late_allowed` variable, which has a default of 3.

```jinja2
and dvce_sent_tstamp <= {{ snowplow_utils.timestamp_add('day', days_late_allowed, 'dvce_created_tstamp') }}
```

#### Filter app IDs

Based on the input provided in the `snowplow__app_id` variable, we filter to only those app ids provided.

```jinja2
and {{ snowplow_utils.app_id_filter(app_ids) }}
```

#### Filter out quarantined sessions

We also exclude any session identifiers listed in the [quarantine manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#quarantine-table) table to avoid processing long running sessions. For a session to be quarantined it must have events spanning longer than the value in your `snowplow__max_session_days` variable.

```jinja2
where session_identifier is not null

    and not exists (select 1 from {{ ref(quarantined_sessions) }} as a where a.session_identifier = e.session_identifier)
```

Once this is all calculated, we merge these with the existing manifest and take the least of the start timestamp, and the greatest of the end timestamp. In the case of a session running over the `snowplow__max_session_days` the end timestamp is re-calculated based on this instead.

### Step 3: Building sessions this run

We next identify which sessions need to be included in the current run, based on our event run limits and the _start_ timestamp of that session. Using the [`return_base_new_event_limits` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/incremental_hooks/return_base_new_event_limits.sql) we get the upper and lower limits as calculated in step 1, but also the lower limit minus the `snowplow__max_session_days` to get the earliest a session could start and still have events included in this run (the session start limit).

We then include in the run any session that:

- Starts after the session start limit
- Starts before or at the upper limit for the run
- Ends after the lower limit for the run

The [base sessions this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#base-sessions-this-run) table then contains any sessions that satisfy these conditions from the lifecycle manifest, but with an `end_tstamp` capped at the upper limit.

### Step 4: Build events this run

Finally we query the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table and filter to events that satisfy the following conditions:

- Session identifier is within the base sessions this run table created in Step 3
- Timestamp is between the min start and max end of sessions in the base sessions this run table created in Step 3
- Timestamp is greater than or equal to the start timestamp for that specific session identifier in base sessions this run table created in Step 3
- (Optional) App ID is in the list provided

This is the [deduplicated](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/) and other transformations are applied such as adding any entities, SDEs, or custom sql.

> **Tip:** The result of all of this is that the range of timestamps in the events this run table will be larger than the range printed in the log, and stored in the base new events limits table, but is required to ensure we correctly process complete sessions and are able to handle late arriving data.

---

# Incremental sessionization logic
> A detailed overview of our incremental run logic using manifest tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/

The general principle behind an incremental model is to identify new events/rows since the previous run of the model, and only process these new events. This minimizes cost and reduces run times. This is great for basic event streams, however you begin to encounter issues for any aggregations you wish to do. To resolve this we treat a `session` as the atomic unit of the majority of our packages, ensuring that when we have a new event for a previously processed session, we have to reprocess all historic events for that session as well as the new events - otherwise our metrics would be incorrect. Note that because we allow [custom session identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) this does not have to be a true web-type `session`.

The logic we have implemented to achieve this is:

1. Identify new events since the previous run of the package
2. Identify all `session_identifier`s associated with the new events
3. Look back over the events table to find all events associated with these `session_identifier`
4. Use only this set of events in all downstream processing

Given the large nature of event tables, Step 3 can be an expensive operation, we solve this issue in 3 ways:

| Problem                                                                                                           | Solution                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| We have to scan the whole events table to find events for a subset of sessions                                    | **Record when any given session started.** We use a [lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#sessions-lifecycle-manifest) table to record the start and end timestamp of all sessions.                                                                                                                                                                                                       |
| Sessions generated by bots can persist for years which would mean scanning years of data every run of the package | **Limit the maximum allowed session length.** We use a [quarantine](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#quarantine-table) table to store the `session_identifier` of any sessions that have exceeded the max allowed session length (`snowplow__max_session_days`). For such sessions, all events are processed up until the max allowed length. Moving forward, no more data is processed for that session. |
| We don't know how up-to-date each model is, so we have to scan each derived table                                 | **Record the max processed timestamp for each model.** We use an [incremental manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#incremental-manifest) table to record the max processed timestamp of all derived tables.                                                                                                                                                                                         |

## Package State

Your Snowplow dbt package can be considered to be in one of 4 states at any given time:

1. First run of the package
2. New model introduced
3. Models out of sync
4. Standard run

This state determines the range of events to start with to process in the next run of the package (note that events earlier than this range may be processed as part of a complete session, as detailed above). We use the `min_last_success` and `max_last_success` for models enabled in the current run, as well as the number of models in the manifest, as the base for our calculation.

**Technical Details**

The identification of which state the package is in is computed by the `get_run_limits` macro which is called in the `snowplow_<package>_base_new_event_limits` model. This macro uses the metadata recorded in `snowplow_<package>_incremental_manifest` to determine the `lower_limit` and `upper_limit` for the run of the package. This limit is based on the calculations shown below, and is based on the field defined in your `snowplow__session_timestamp` variable.

To get the min and max last success, we run the following query:

```sql
select

    min(last_success) as min_last_success,

    max(last_success) as max_last_success,

    coalesce(count(*), 0) as models

from snowplow_<package>_incremental_manifest

where model in (array_of_snowplow_tagged_enabled_models)
```

Based on these the model is in one of the four states.

> **Tip:** In all states the `upper_limit` is limited by the `snowplow__backfill_limit_days` variable. This protects against back-fills with many rows causing very long run times.

### State 1: First run of the package

If there are no enabled models already in the manifest then we process from the start date up to the backfill limit or now, whichever is older:

**`lower_limit`**: `snowplow__start_date` **`upper_limit`**: `least(current_tstamp, snowplow__start_date + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    snowplow__start_date :crit, m1, 2020-01-01, 2m

    section Unprocessed
    Future Runs :crit, c, after a1, 2d

    section Processing
    snowplow__backfill_limit_days :active, a1, 2020-01-01, 1d

    section Processed
```

### State 2: New model introduced

If there are enabled models that aren't in the manifest table then a new model tagged with `snowplow_<package>_incremental` has been added since the last run; this can happen with a new custom model, or you have enabled some previously disabled custom modules. In this case the package will replay all previously processed events in order to back-fill the new model.

**`lower_limit`**: `snowplow__start_date` **`upper_limit`**: `least(max_last_success, snowplow__start_date + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    snowplow__start_date :crit, m1, 2020-01-01, 2m

    section Unprocessed
    Future Runs (new model) :crit, c, after a1 , 2d

    section Processing
    snowplow__backfill_limit_days :active, a1, 2020-01-01, 1d

    section Processed
    Other models :c, 2020-01-01, 3d
```

### State 3: Models out of sync

If the `min_last_success` is less than the `max_last_success` it means the tagged models are out of sync, for example due to a particular model failing to execute successfully during the previous run or as part of catching up on a new model. The package will attempt to sync all models as far as your backfill limit will allow.

**`lower_limit`**: `min_last_success - snowplow__lookback_window_hours` **`upper_limit`**: `least(max_last_success, min_last_success + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    min_last_success :crit, m1, 2020-01-02, 5m

    section Unprocessed
    Future Runs (out of sync model) :crit, c, after a2, 2d

    section Processing
    snowplow__lookback_window_hours :active, a1, 2020-01-01 18:00:00, 6h
    snowplow__backfill_limit_days :active, a2, after b1, 1d

    section Processed
    Out of sync model :b1, 2020-01-01, 1d
    Other models :b2, 2020-01-01, 4d
```

### State 4: Standard run

If none of the above criteria are met, then we consider it a 'standard run' where all models are in sync and we carry on from the last processed event.

**`lower_limit`**: `max_last_success - snowplow__lookback_window_hours` **`upper_limit`**: `least(current_tstamp, max_last_success + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    max_last_success :crit, m1, 2020-01-04, 8m

    section Unprocessed
    Future Runs :crit, c, after a2 , 2d

    section Processing
    snowplow__lookback_window_hours :active, a1, 2020-01-03 18:00:00, 6h
    snowplow__backfill_limit_days :active, a2, after b1, 1d

    section Processed
    Previous Runs :b1, 2020-01-01, 3d
```

> **Note:** Remember, this upper and lower limit is only used to identify which events are considered to get the session identifiers for the run, additional events outside this date range may be included to ensure that full sessions are processed. See [late arriving data](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/) for more information.

### How to identify the current state

If you want to check the current state of a model, run the `snowplow_<package>_base_new_event_limits` model. This will log the current state to the CLI while causing no disruption to the incremental processing of events. You will also see this in the logs of any incremental run.

```bash
dbt run --select snowplow_<package>_base_new_event_limits

...

00:26:29 + Snowplow: Standard incremental run

00:26:29 + Snowplow: Processing data between 2021-01-05 17:59:32 and 2021-01-07 23:59:32
```

- State 1: `Snowplow: No data in manifest. Processing data from start_date`
- State 2: `Snowplow: New Snowplow incremental model. Backfilling`
- State 3: `Snowplow: Snowplow incremental models out of sync. Syncing`
- State 4: `Snowplow: Standard incremental run`

> **Note:** In all states, although much more likely in state 1, it is possible that a run finds no incremental data to process. In this case you will see a warning in the logs of the form `Snowplow Warning: No data in <table> for date range from variables, please modify your run variables to include data if this is not expected.`. In this case you should increase your start date if no runs have completed, or increase your backfill limit days if there is a gap in your data before the next records.

In reality, the events in your [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table will have a wider range of timestamps than those calculated here. For more details see the [full run calculation](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/full-run-calculation/) page.

---

# Snowplow dbt package mechanics
> Key mechanics of our packages
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/

The following pages contain information relating to the core logic and processing of our packages, it is not required to understand these concepts to make use of our packages but they are provided for more advanced users who may wish to produce more complex [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) or better understand what's going on under the hood of the packages. We refer to these elements throughout the docs.

> **Tip:** Throughout the packages and these docs, we will refer to a `session`, or `session_identifier`. This is the atomic unit of re-processing, as described in the [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) and by default usually refers to the actual session identifier within your tracker; however this language is in part only used for historic reasons. Thanks to the [custom identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) in our packages, these _sessions_ can refer to any identifier you define and our packages will ensure that all events (within the limits) are processed in a single run. A similar case applies to user identifiers as well.

---

# Late arriving data
> A detailed overview of how the data flows and how to manage variables for handling edge cases.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/

```mermaid
graph LR
    style A color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style B color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style C color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style D color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#FEEEBD


    A[event created] -->B[event sent to Snowplow]
    B[event sent to Snowplow] --> C[event received by collector]
    C[event received by collector] --> D[event loaded to events table]
```

Normally, there is very little delay between each step of the dataflow, dependent on your pipeline setup and other environmental conditions.

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style C color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style D color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#633EB5


    A[dvce_created_tstamp] -->|millisecond| B[dvce_sent_tstamp]
    B[dvce_sent_tstamp] --> |millisecond|C[collector_tstamp]
    C[collector_tstamp] --> |millisecond|D[load_tstamp]
```

However there can be times when data arrives "late", which can mean different things depending on where the delay in the flow of data occurs. Here we cover the 2 most common.

### Late Sent Events

The most common scenario is that the device the event is generated on goes offline, leading to a larger than usual gap between the `dvce_created_tstamp` and the `dvce_sent_tstamp`:

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style C color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style D color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#633EB5


    A[dvce_created_tstamp] -->|HOURS| B[dvce_sent_tstamp]
    B[dvce_sent_tstamp] --> |millisecond|C[collector_tstamp]
    C[collector_tstamp] --> |millisecond|D[load_tstamp]
```

In our packages, processing this type of data is managed by the `snowplow__days_late_allowed` variable. Any event with a delay greater than this value between being created and being sent will not be processed. However, assuming the data lands within this limit the whole session will be reprocessed to include this event.

### Late Loaded Events

The other most common case is that of late loaded events, where the gap is between the collector and load timestamps. This may be due to batch loading, or issues with your pipeline.

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style C color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style D color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#633EB5


    A[dvce_created_tstamp] -->|millisecond| B[dvce_sent_tstamp]
    B[dvce_sent_tstamp] --> |millisecond|C[collector_tstamp]
    C[collector_tstamp] --> |HOURS|D[load_tstamp]
```

Because by default our packages uses `collector_tstamp` to identify new data, this can mean that these events are not picked up (and therefore the sessions not reprocessed to include them). If you are using `load_tstamp` as your `snowplow__session_timestamp` then this is not an issue.

To manage this we use the `snowplow__lookback_window_hours` variable to add a _buffer_ to the previous maximum processed timestamp to account for some events being delayed. In the case of batch loading, you should ensure this window is at least a few hours greater than your batch period.

Note that this isn't an issue if _all_ events are delayed, because if all events are delayed the previous maximum processed timestamp would still be before any of these events. In the event of the model running part way through the loading of a set of out-of-order events (e.g. an event recovery), you can increase the `snowplow__lookback_window_hours` to forcibly reprocess a larger window of data.

For more details on how all these variables combine, please see the docs on the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of our packages.

---

# Manifest tables
> Details around the manifest tables we use in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/

Each of our packages has a set of manifest tables that manage the [Incremental Sessionization Logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) logic of our package, as well as quarantining long running sessions.

> **Note:** These manifest tables are critical to the package **and as such are protected from full refreshes, i.e. being dropped, when running in production by default**. In development refreshes are enabled.

The `allow_refresh()` macro defines the protection behavior. As [dbt recommends](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles#understanding-targets-in-profiles), target names are used here to differentiate between your prod and dev environment. By default, this macro assumes your dev target is named `dev`. This can be changed by setting the `snowplow__dev_target_name` var in your `dbt_project.yml` file.

To full refresh any of the manifest models in production as part of a `--full-refresh`, set the `snowplow__allow_refresh` to `true` at run time.

Alternatively, you can amend the behavior of this macro entirely by overwriting it. See the [Overwriting Macros](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/) section for more details.

## Incremental manifest

The majority of our packages have an incremental manifest table; by default this is in your `_snowplow_manifest` suffixed schema, and will have the name `snowplow_<package_name>_incremental_manifest`. This table exists to track the [state](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#how-to-identify-the-current-state) of each of the models in the package, including any [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) that have been tagged.

This table has 2 columns:

- `model`: The name of the model
- `last_success`: The timestamp of the max last event processed by the model, based on the field defined in the `snowplow__session_timestamp` variable

As this is the source of truth for processing for the package, it is highly recommended to never alter or edit this table directly as this can lead to unexpected outcomes. The manifest table is updated as part of an `on-run-end` hook, which calls the `snowplow_incremental_post_hook()` macro.

## Sessions lifecycle manifest

The majority of our packages have a session lifecycle manifest table; by default this is in your `_snowplow_manifest` suffixed schema, and will have the name `snowplow_<package_name>_base_sessions_lifecycle_manifest`. This table exists to track the start and end timestamps of any given session, allowing us to avoid a full table scan each run.

This table has 4 columns:

- `session_identifier`: The unique identifier for the session
- `user_identifier`: The unique user identifier for the session. Note that if multiple user identifiers exist for a given session we only take one.
- `start_tstamp`: The timestamp of the first event to process for the session (note due to late arriving data, this may not actually be the true first event in rare cases)
- `end_tstamp`: The timestamp of the last event to process for the session (note this may be capped with the `snowplow__max_session_days` variable)

Timestamps based on the field defined in the `snowplow__session_timestamp` variable. This table is used to calculate which sessions are to be processed in a run, and also helps inform the run limits.

## Quarantine table

Many of our packages have a quarantine table; by default this is in your `_snowplow_manifest` suffixed schema, and will have the name `snowplow_<package_name>_base_quarantined_sessions`. This table exists to track sessions that have gone beyond the `snowplow__max_session_days` limit and avoid re-processing them again.

This table has 1 column:

- `session_identifier`: The unique identifier for the session to not process in any further run.

This table is updated on the post-hook of the [base sessions this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#base-sessions-this-run) table. If there are any additional sessions you identify you want to remove from processing, you can manually add them to this table but ensure they do not already exist.

---

# Snowplow optimized materialization
> Details on our optimized version of the dbt incremental materialization and how to enable it.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/

> **Tip:** This functionality requires dispatching to our macros over dbt core, see how to do this on the [dispatch setup](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/dispatch/) page.

Our packages make use of the standard dbt `incremental` [materialization](https://docs.getdbt.com/docs/build/materializations#incremental) with an optimization applied for incremental models. The advantage is that it limits table scan on the target table when updating/inserting based on the new data. This improves performance and reduces cost. We do this by overriding the macro that generates the sql for the `merge` and `insert_delete` incremental methods.

All other features of the `incremental` materialization are supported including `incremental_predicates` and `on_schema_change`. The code for the overridden macro can be found [on GitHub](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/materializations/base_incremental/common/get_merge_sql.sql).

## Usage

### Controlling the buffer size

We calculate the upper and lower limit from the source table (usually a [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/) table) before adding a buffer to the lower limit to cover late arriving data or other issues with specific timestamps. This buffer is controlled by the `snowplow__upsert_lookback_days` variable and usually has a default of 30.

To disable this buffer entirely, you can either set `snowplow__upsert_lookback_days` to 0 or you can `disable_upsert_lookback` to `true` in your \*\* model config\*\* if you want to do this for a specific model. **Note this is a [model config](https://docs.getdbt.com/reference/model-configs), not a variable.**

### Adding to your own models

All our incremental models have this functionality enabled by default, but if you wish to use it in a [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) you you need to ensure a `unique_key` and `upsert_date_key` are provided in the model config, and that `snowplow_optimize=true` in the config as well, e.g.

```jinja2
{{

  config(

    materialized='incremental',

    unique_key='page_view_id',

    upsert_date_key='start_tstamp',

    snowplow_optimize = true,

    ...

  )

}}
```

## Disabling the optimization

To disable the optimized upsert entirely and use the default incremental materialization, set the `snowplow_optimize` to `false` in your model config. **Note this is a [model config](https://docs.getdbt.com/reference/model-configs), not a variable.**

---

# Overridable macros in Snowplow dbt packages
> Override and customize default macros in Snowplow dbt packages to extend package functionality.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/

Many of our packages are built using [macros](https://docs.getdbt.com/docs/build/jinja-macros) to allow easier support of multiple warehouses. Some of these macros are designed to be overridable to give an easy route to customization for the user. You can find a list of the overridable macros for each package in the [package details](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) pages.

The easiest way to override any given macro is to create a `default__` version within your dbt project, for example, to change the `filter_bots` macro in the `unified` package you would have:

```jinja2
{% macro default__filter_bots(table_alias = none) %}

    and {% if table_alias %}{{table_alias~'.'}}{% endif %}useragent not similar to '%(YOUR_CUSTOM_PATTERN|bot|crawl|slurp|spider|archiv|spinn|sniff|seo|audit|survey|pingdom|worm|capture|(browser|screen)shots|analyz|index|thumb|check|facebook|PingdomBot|PhantomJS|YandexBot|Twitterbot|a_archiver|facebookexternalhit|Bingbot|BingPreview|Googlebot|Baiduspider|360(Spider|User-agent)|semalt)%'

{% endmacro %}
```

> **Tip:** Make sure to check the source code for the macro to understand what arguments are required and make sure you start from the version applicable to your warehouse.

---

# This run tables
> Details about the this run table approach use in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/

Our packages make use of a series of `this run` tables, named such because they are dropped and recreated each `dbt run` to only contain relevant information that that run. These tables allow for efficient re-use of information throughout a single run, and also allow you to debug any issues should a run fail.

## Events this run

As described in the [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), the first thing we do in any given run is identify what events need to be processed as part of that run. These events are [deduplicated](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/), have relevant [entities and SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) extracted and attached, and the appropriate [identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) are calculated.

This table of events is then used for all further processing in the package, we refer to this table as `events this run`. This is the table to use in the case of building most [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) that require the event level data.

In all cases where this table exists, it is in the `models/base/scratch/` directory within the package.

| Package     | Events This Run Name                              |
| ----------- | ------------------------------------------------- |
| unified     | `snowplow_unified_events_this_run`                |
| ecommerce   | `snowplow_ecommerce_base_events_this_run`         |
| media       | `snowplow_media_player_base_events_this_run`      |
| normalize   | `snowplow_normalize_base_events_this_run`         |
| attribution | None, this package does not build from raw events |

> **Tip:** Unified also contains a `snowplow_unified_base_events_this_run`, however this builds into `snowplow_unified_events_this_run` so we recommend you use the more complete table.

## Base sessions this run

The sessions this run table is built before the events this run table, and contains a list of all all the sessions that are being processed in the current run. It is a modified sub-set of the [sessions lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) with only the sessions within the run limits, and with the end timestamp capped to the upper limit of the run (in case of historic re-processing).

In all packages that contain this table, it named of the form `snowplow_<package_name>_base_sessions_this_run` and is located in the `models/base/scratch` directory within the package.

There should be no need to directly interact with this table but may be useful debugging any errors.

## Other this run tables

Our packages are built in a way that makes it easy to make small alterations and additions to the derived models. Most derived models, such as `sessions`, or `views` will have a corresponding this run table. The actual incremental derived model itself is just a `select *` from this table (with a date field added for Databricks). This becomes useful in [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) and is also helpful to run tests on just the incremental data or to identify any issues. We recommend using the output of these where possible, but not directly overwriting them as this may put you out of sync with any updates we release. See the [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) docs for more information about how best to use these tables.

The usual folder structure, although this can vary a bit by package is that these this run tables tend to sit at either `models/<module>/scratch/snowplow_<package_name>_<module>_this_run` or `models/<module>/scratch/<warehouse>/snowplow_<package_name>_<module>_this_run` when there are different versions of the model per warehouse.

---

# Model your data with SQL Runner (legacy)
> Guides for setting up the legacy SQL Runner and running our web and mobile models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/

> **Tip:** For any new developments we highly recommend using our [dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/) instead. SQL Runner is no longer under active development and will only receive bug fixes in the future.

[SQL Runner](https://github.com/snowplow/sql-runner) enables you to execute SQL scripts against the Snowplow data in your data warehouse. Specifically, it allows you to organize your SQL scripts in templatable playbooks, and execute them in series or in parallel on Snowflake, Amazon Redshift, GCP BigQuery and PostgreSQL.

A SQL Runner data model consists of:

- SQL files (containing one or more SQL statements)
- Playbooks (YAML files organizing the SQL into steps)

### Available models

| Model                                                                                               | Redshift | BigQuery | Snowflake |
| --------------------------------------------------------------------------------------------------- | -------- | -------- | --------- |
| [Web](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/)       | 1.3.1    | 1.0.4    | 1.0.2     |
| [Mobile](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-mobile-data-model/) | 1.1.0    | 1.1.0    | 1.1.0     |

### Playbooks

A playbook consists of one of more _steps_, each of which consists of one or more _queries_. Steps are run in series, queries are run in parallel within the step.

Each query contains the path to a _query file_. 

All steps are applied against all _targets_. All targets are processed in parallel.

In the following example, a.sql, b.sql and c.sql are run in parallel.

```yaml
:steps:

  - :name: "Run a,b and c in parallel"

    :queries:

      - :name: a

        :file: a.sql

      - :name: b

        :file: b.sql

      - :name: c

        :file: c.sql
```

By contrast, in the example below, the three SQL files are executed in sequence.

```yaml
:steps:

  - :name: "Run a..."

    :queries:

      - :name: a

        :file: a.sql

  - :name: "...then run b..."

    :queries:

      - :name: b

        :file: b.sql

  - :name: "...then run c..."

    :queries:

      - :name: c

        :file: c.sql
```

Playbooks can be templated, and corresponding variables can be passed in with the var flag like this:

```bash
sql-runner -var host=value,username=value2,password=value3
```

Here is the corresponding playbook template:

```yaml
:targets:

- :name: "My Postgres database 1"

  :type: postgres

  :host: {{.host}}

  :database: sql_runner_tests_1

  :port: 5432

  :username: {{.username}}

  :password: {{.password}}

  :ssl: false # SSL disabled by default

:variables:

  :test_schema: sql_runner_tests

  :timeFormat: "2006_01_02"

:steps:

- :name: Create schema and table

  :queries:

    - :name: Create schema and table

      :file: postgres-sql/good/1.sql

      :template: true
```

### SQL files

A query file contains one or more SQL statements. These are executed "raw" (i.e. not in a transaction) in series by SQL Runner. If the query file is flagged as a _template_ in the playbook, then the file is pre-processed as a template before being executed.

**Note**: If your query is a template that requires pre-processing, you must add `template: true` to the query definition in the playbook yml file, for example:

```yaml
:name: "Run a.."

    :queries:

      - :name: a

        :file: a.sql

        :template: true
```

### Templates

Templates are run through Golang's [text template processor](https://pkg.go.dev/text/template). The template processor can access all _variables_ defined in the playbook.

The following custom functions are also supported:

- `nowWithFormat [timeFormat]`: where `timeFormat` is a valid Golang [time format](http://golang.org/pkg/time/#Time.Format)
- `systemEnv "ENV_VAR"`: where `ENV_VAR` is a key for a valid environment variable
- `awsEnvCredentials`: supports passing credentials through environment variables, such as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
- `awsProfileCredentials`: supports getting credentials from a credentials file, also used by boto/awscli
- `awsEC2RoleCredentials`: supports getting role-based credentials, i.e. getting the automatically generated credentials in EC2 instances
- `awsChainCredentials`: tries to get credentials from each of the three methods above in order, using the first one returned
- `randomInt`: will return a random integer

**Note**: All AWS functions output strings in the Redshift credentials format (`CREDENTIALS 'aws_access_key_id=%s;aws_secret_access_key=%s'`).

For an example query file using templating see: [integration/resources/postgres-sql/good/3.sql](https://github.com/snowplow/sql-runner/blob/master/integration/resources/postgres-sql/good/3.sql)

### Failure modes

If a statement fails in a query file, the query will terminate and report failure.

If a query fails, its sibling queries will continue running, but no further steps will run.

Failures in one target do not affect other targets in any way.

### Return codes

```text
- 0 for no errors

- 5 for target initialization errors

- 6 for query errors

- 7 for both types of error

- 8 for no queries run
```

### Target configuration

#### Redshift

If your storage target is Amazon Redshift, then the target configuration in the playbook is:

```yaml
targets:

  - name: "My Redshift database"

    type: redshift

    host: ADD HERE # The endpoint as shown in the Redshift console

    database: ADD HERE # Name of database

    port: 5439 # Default Redshift port

    username: ADD HERE

    password: ADD HERE

    ssl: false # SSL disabled by default

variables:

  ...
```

#### BigQuery

To access a BigQuery project, sql-runner will need some Google credentials. These can be set up by creating a new service account in the GCP console, then providing its private key to the application via a GOOGLE\_APPLICATION\_CREDENTIALS environment variable - a detailed walkthrough of this process is available on the [GCP documentation website](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually).

After the credentials are set up, simply create a playbook with the following BigQuery-specific target configuration:

```yaml
targets:

  - name: "My BigQuery database"

    type: bigquery

    project: ADD HERE # Project ID as shown in the GCP console's front page

variables:

    ...
```

#### Snowflake

If your data warehouse is Snowflake, then the SQL Runner playbooks will have a target configuration as:

```yaml
targets:

  - name: "My Snowflake database"

    type: snowflake

    account: ADD HERE # Your Snowflake account name

    database: ADD HERE # Name of database

    warehouse: ADD HERE # Name of warehouse to run the queries

    username: ADD HERE

    password: ADD HERE

    host: # Leave blank

    port: # Leave blank

    ssl: true # Snowflake connection is always secured by TLS

    query_tag: ADD HERE # optional, available since v0.10.0

variables:

    ...
```

The `query_tag` parameter sets the `QUERY_TAG` [session parameter](https://docs.snowflake.com/en/sql-reference/parameters.html#query-tag) in Snowflake. When set, it will be applied to all queries included in the playbook.

#### PostgreSQL

Finally, if your storage target is PostgreSQL, then can be configured as:

```yaml
targets:

  - name: "My Postgres database"

    type: postgres

    host: ADD HERE

    database: ADD HERE # Name of database

    port: 5432 # Default Postgres port

    username: ADD HERE

    password: ADD HERE

    ssl: false # SSL disabled by default

variables:
```

That's it - you're now ready to start running SQL against your data warehouse!

## User guide

SQL Runner is a zero-dependency binary and can be found as a [release asset](https://github.com/snowplow/sql-runner/releases/latest) for:

- [macOS](https://github.com/snowplow/sql-runner/releases/download/0.10.1/sql_runner_0.10.1_darwin_amd64.zip)
- [Linux](https://github.com/snowplow/sql-runner/releases/download/0.10.1/sql_runner_0.10.1_linux_amd64.zip)
- [Windows](https://github.com/snowplow/sql-runner/releases/download/0.10.1/sql_runner_0.10.1_windows_amd64.zip)

### CLI Arguments

```text
./sql-runner --help

sql-runner version: 0.10.1

Run playbooks of SQL scripts in series and parallel on Redshift, Postgres, BigQuery and Snowflake

Usage:

-checkLock string

  	Checks whether the lockfile already exists

-consul string

  	The address of a consul server with playbooks and SQL files stored in KV pairs

-consulOnlyForLock

  	Will read playbooks locally, but use Consul for locking.

-deleteLock string

  	Will attempt to delete a lockfile if it exists

-dryRun

  	Runs through a playbook without executing any of the SQL

-fillTemplates

  	Will print all queries after templates are filled

-fromStep string

  	Starts from a given step defined in your playbook

-help

  	Shows this message

-lock string

  	Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully

-playbook string

  	Playbook of SQL scripts to execute

-runQuery string

  	Will run a single query in the playbook

-showQueryOutput

  	Will print all output from queries

-softLock string

  	Optional argument, like '-lock' but the lockfile will be deleted even if the run fails

-sqlroot string

  	Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")

-var value

  	Variables to be passed to the playbook, in the key=value format

-version

  	Shows the program version
```

#### More on Consul

Using the `-consul` argument results in the following changes:

- The `-playbook` argument becomes the key that is used to look for the playbook in Consul.
- The `-sqlroot` argument also becomes a key argument for Consul.
- The `-lock` argument creates a lock as a Consul key value pair
- The `-softLock` argument creates a lock as a Consul key value pair
- The `-checkLock` argument searches in Consul for a lock
- The `-deleteLock` argument searches in Consul for a lock

If you pass in the default:

- `./sql-runner -consul "localhost:8500" -playbook "sql-runner/playbook/1"`

This results in:

- Looking for your playbook file at this key `sql-runner/playbook/1`
- Expecting all your SQL file keys to begin with `sql-runner/playbook/<SQL path from playbook>`

However as the key here can be used as a both a data and folder node we have added a new sqlroot option:

- `./sql-runner -consul "localhost:8500" -playbook "sql-runner/playbook/1" -sqlroot PLAYBOOK_CHILD`

This results in:

- Looking for your playbook file at this key `sql-runner/playbook/1`
- Expecting all your SQL file keys to begin with `sql-runner/playbook/1/<SQL path from playbook>`
  - The data node is used as a folder node as well.

---

# Migrating from SQL Runner to dbt
> SQL Runner is no longer actively developed and users should try to migrate to dbt where possible, this guide helps you do that.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/migrating-to-dbt/

> **Info:** This guide assumes you are running the standard web and/or mobile SQL Runner models, if you have built any custom models or customized the standard models in any way, you will need to make these same changes to the dbt models. See our section on [custom dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) for more information on how to do this.

## Why Migrate?

SQL Runner is currently in maintenance mode, while we will continue to fix bugs when they are identified, we are not actively developing the tool or the models anymore and at some point in the future may deprecate it entirely. Our dbt models on the other hand are under active development, with new features and optimizations being made regularly. It is also a more widely used tool, meaning installation and management is far easier (or you can use tools like dbt Cloud, or our CDI customers can run dbt models the same way you can SQL Runner). We also have a far wider range of packages available in dbt including ecommerce, marketing attribution, and a package to normalize your Snowplow data.

In dbt we also support Databricks and Postgres warehouses in addition to Snowflake, BigQuery, and Redshift. Newer tracking plugins are only being modeled within our dbt packages.

In general, if you are happy with SQL Runner and don't foresee a need to add more models in the future then there is no need to migrate; however if you are starting from scratch, or would like to make use of our wider range of models, then you should consider migrating to dbt.

## Differences between the tools

The core of the web and mobile models (e.g. page/screen views, sessions, and users) are the same across SQL Runner and dbt, however the dbt versions contain some additional fields which may be useful for analysis (such as conversions, event counts, and human-readable language information in the web package), as well as additional optional modules for web such as consent and core web vitals. Below this the logic used to process the data is roughly the same, although we have made some optimizations and added more flexibility in how this processing is done in dbt.

We recommend you take a look at the [docs](/docs/modeling-your-data/modeling-your-data-with-dbt/) for our dbt packages to get a better understanding of how they work and how you can use them going forward.

## Pre-requisites

We assume that you have dbt installed, a working connection, and some basic understanding of using dbt including installing packages and running models.

## Mapping the variables

While the variables for your SQL Runner models are spread throughout the files, in dbt all variables are in your `dbt_project.yml` file. We have mostly been consistent between the two tools, with the dbt variables being prefixed by `snowplow__`, but some have new names. The table below maps each SQL Runner variable to the equivalent dbt variable, but there are many more you can set to customize how the models run - you can read about these in the relevant [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) page.

> **Tip:** When using multiple dbt packages you must be careful to specify which scope a variable or configuration is defined within. In general, always specify each value in your `dbt_project.yml` nested under the specific package e.g.
>
> ```yml
> vars:
>
>   snowplow_web:
>
>     snowplow__atomic_schema: schema_with_snowplow_web_events
>
>   snowplow_mobile:
>
>     snowplow__atomic_schema: schema_with_snowplow_mobile_events
> ```
>
> You can read more about variable scoping in dbt's docs around [variable precedence](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#variable-precedence).

> Values in bold have a different name instead of just the prefix

| SQL Runner Variable                    | dbt variable                                                                                            |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **`app_errors`**                       | **`snowplow__enable_app_errors_module`**                                                                |
| **`app_id_filters`**                   | **`snowplow__app_id`**                                                                                  |
| **`application_context`**              | **`snowplow__enable_application_context`**                                                              |
| `cleanup_mode`                         | No equivalent variable _(closest is `snowplow__allow_refresh` combined with dbt `--full-refresh` flag)_ |
| `cluster_by`                           | No equivalent variable _(Clustering defined in model)_                                                  |
| `days_late_allowed`                    | `snowplow__days_late_allowed`                                                                           |
| `derived_tstamp_partitioned`           | `snowplow__derived_tstamp_partitioned`                                                                  |
| **`enabled`** (Mobile app errors only) | **`snowplow__enable_app_errors_module`**                                                                |
| `ends_run`                             | No equivalent variable                                                                                  |
| `entropy`                              | No equivalent variable                                                                                  |
| **`geolocation_context`**              | **`snowplow__enable_geolocation_context`**                                                              |
| `heartbeat`                            | `snowplow__heartbeat`                                                                                   |
| **`iab`**                              | **`snowplow__enable_iab`**                                                                              |
| **`input_schema`**                     | **`snowplow__atomic_schema`**                                                                           |
| `lookback_window_hours`                | `snowplow__lookback_window_hours`                                                                       |
| **`minimumVisitLength`**               | **`snowplow_min_visit_length`**                                                                         |
| **`mobile_context`**                   | **`snowplow__enable_mobile_context`**                                                                   |
| **`model_version`**                    | No equivalent variable                                                                                  |
| `output_schema`                        | Set in `models` part of project file, see relevant configuration page for more info.                    |
| **`platform_filters`**                 | **`snowplow__platform`**                                                                                |
| `scratch_schema`                       | Set in `models` part of project file, see relevant configuration page for more info.                    |
| **`screen_context`**                   | **`snowplow__enable_screen_contextt`**                                                                  |
| `session_lookback_days`                | `snowplow__session_lookback_days` _(default increased to 730)_                                          |
| `skip_derived`                         | No equivalent variable _(use dbt `--select` flag)_                                                      |
| `stage_next`                           | No equivalent variable                                                                                  |
| `start_date`                           | `snowplow__start_date`                                                                                  |
| `ua_bot_filter`                        | `snowplow__ua_bot_filter`                                                                               |
| **`ua_parser`**                        | **`snowplow__enable_ua`**                                                                               |
| **`update_cadence_days`**              | **`snowplow__backfill_limit_days`** _(default increased to 30)_                                         |
| `upsert_lookback_days`                 | `snowplow__upsert_lookback_days`                                                                        |
| **`yauaa`**                            | **`snowplow__enable_yauaa`**                                                                            |

## Setting up and running our dbt packages

The latest information for our packages can be found in the [quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) section of our docs. It is highly recommended you just start running your dbt project from scratch and process all data from the start date to ensure the package works as intended and all your data is processed in the same way.

## Migrate existing derived data

> **Note:** The dbt package uses slightly different logic for processing, including the quarantining of sessions and different format manifest tables. It is highly recommended that you just run the dbt project from your start date. The following is a best-effort suggestion and we make no guarantee that all data will be correctly processed or that issues may not happen later in the lifetime of the project.
>
> This method will also not correctly populate the user stitching table or process user stitching for historic data.

There may be cases where running the dbt models from scratch is not a viable option for you, in this case it is possible to migrate your existing derived SQL Runner data into the derived tables produced by dbt, however this will result in your data being generated from two slightly differing logics.

It is advisable to produce your dbt tables into new schemas where possible, even though the derived tables should have different names; this will help keep your data separate and ensure that as we go through the following steps that dbt does not overwrite your SQL Runner tables.

> **Note:** Postgres only supports the `MERGE` statement in version 15 and up, if you are using an older version you will need to alter the commands to be a `DELETE`+`INSERT` instead.

### Create dbt tables by doing a recent-dated run

Because of the difference in manifest tables and incremental logic between SQL Runner and dbt models it makes sense to first create the dbt tables and then insert your existing data into them, rather than try and create the dbt tables directly from your SQL Runner data.

Once you have your dbt project and variables set up, change your `snowplow__start_date` to a recent date, say 7 days before the end of your last SQL Runner processed date, and run the project once. This will produce all the dbt tables including the manifest tables needed to manage the incremental logic, and ensure a good overlap between the end of your SQL Runner processing and the start of dbt processing.

### Merge your existing data into the new tables (web)

The following SQL will merge the existing web records in your SQL Runner derived tables into the new dbt derived tables, please run each one as required. This approach will leave any columns that only exist in dbt `null`. Please ensure you replace the schema and table names with your ones where appropriate.

**If you are using Redshift, be sure to `commit` your changes.**

> **Note:** It is possible, particularly for columns which may have been null, that the types of columns across the two tables don't entirely match. Your warehouse may manage this for you, or you may have to use a `cast(col_name as new_type)` in place of just selecting the column based on any error message you receive.

#### Page Views

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_web_page_views t

USING <SQL_RUNNER_DERIVED_SCHEMA>.page_views s

ON T.page_view_id = s.page_view_id

WHEN NOT MATCHED THEN

INSERT

(

    page_view_id,

    event_id,

    app_id,

    user_id,

    domain_userid,

    network_userid,

    domain_sessionid,

    domain_sessionidx,

    page_view_in_session_index,

    page_views_in_session,

    dvce_created_tstamp,

    collector_tstamp,

    derived_tstamp,

    start_tstamp,

    end_tstamp,

    engaged_time_in_s,

    absolute_time_in_s,

    horizontal_pixels_scrolled,

    vertical_pixels_scrolled,

    horizontal_percentage_scrolled,

    vertical_percentage_scrolled,

    doc_width,

    doc_height,

    page_title,

    page_url,

    page_urlscheme,

    page_urlhost,

    page_urlpath,

    page_urlquery,

    page_urlfragment,

    mkt_medium,

    mkt_source,

    mkt_term,

    mkt_content,

    mkt_campaign,

    mkt_clickid,

    mkt_network,

    page_referrer,

    refr_urlscheme,

    refr_urlhost,

    refr_urlpath,

    refr_urlquery,

    refr_urlfragment,

    refr_medium,

    refr_source,

    refr_term,

    geo_country,

    geo_region,

    geo_region_name,

    geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_timezone,

    user_ipaddress,

    useragent,

    br_lang,

    br_viewwidth,

    br_viewheight,

    br_colordepth,

    br_renderengine,

    os_timezone,

    category,

    primary_impact,

    reason,

    spider_or_robot,

    useragent_family,

    useragent_major,

    useragent_minor,

    useragent_patch,

    useragent_version,

    os_family,

    os_major,

    os_minor,

    os_patch,

    os_patch_minor,

    os_version,

    device_family,

    device_class,

    agent_class,

    agent_name,

    agent_name_version,

    agent_name_version_major,

    agent_version,

    agent_version_major,

    device_brand,

    device_name,

    device_version,

    layout_engine_class,

    layout_engine_name,

    layout_engine_name_version,

    layout_engine_name_version_major,

    layout_engine_version,

    layout_engine_version_major,

    operating_system_class,

    operating_system_name,

    operating_system_name_version,

    operating_system_version

)

VALUES

(

    s.page_view_id,

    s.event_id,

    s.app_id,

    s.user_id,

    s.domain_userid,

    s.network_userid,

    s.domain_sessionid,

    s.domain_sessionidx,

    s.page_view_in_session_index,

    s.page_views_in_session,

    s.dvce_created_tstamp,

    s.collector_tstamp,

    s.derived_tstamp,

    s.start_tstamp,

    s.end_tstamp,

    s.engaged_time_in_s,

    s.absolute_time_in_s,

    s.horizontal_pixels_scrolled,

    s.vertical_pixels_scrolled,

    s.horizontal_percentage_scrolled,

    s.vertical_percentage_scrolled,

    s.doc_width,

    s.doc_height,

    s.page_title,

    s.page_url,

    s.page_urlscheme,

    s.page_urlhost,

    s.page_urlpath,

    s.page_urlquery,

    s.page_urlfragment,

    s.mkt_medium,

    s.mkt_source,

    s.mkt_term,

    s.mkt_content,

    s.mkt_campaign,

    s.mkt_clickid,

    s.mkt_network,

    s.page_referrer,

    s.refr_urlscheme,

    s.refr_urlhost,

    s.refr_urlpath,

    s.refr_urlquery,

    s.refr_urlfragment,

    s.refr_medium,

    s.refr_source,

    s.refr_term,

    s.geo_country,

    s.geo_region,

    s.geo_region_name,

    s.geo_city,

    s.geo_zipcode,

    s.geo_latitude,

    s.geo_longitude,

    s.geo_timezone,

    s.user_ipaddress,

    s.useragent,

    s.br_lang,

    s.br_viewwidth,

    s.br_viewheight,

    s.br_colordepth,

    s.br_renderengine,

    s.os_timezone,

    s.category,

    s.primary_impact,

    s.reason,

    s.spider_or_robot,

    s.useragent_family,

    s.useragent_major,

    s.useragent_minor,

    s.useragent_patch,

    s.useragent_version,

    s.os_family,

    s.os_major,

    s.os_minor,

    s.os_patch,

    s.os_patch_minor,

    s.os_version,

    s.device_family,

    s.device_class,

    s.agent_class,

    s.agent_name,

    s.agent_name_version,

    s.agent_name_version_major,

    s.agent_version,

    s.agent_version_major,

    s.device_brand,

    s.device_name,

    s.device_version,

    s.layout_engine_class,

    s.layout_engine_name,

    s.layout_engine_name_version,

    s.layout_engine_name_version_major,

    s.layout_engine_version,

    s.layout_engine_version_major,

    s.operating_system_class,

    s.operating_system_name,

    s.operating_system_name_version,

    s.operating_system_version

);
```

#### Sessions

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_web_sessions t

USING <SQL_RUNNER_DERIVED_SCHEMA>.sessions s

ON T.domain_sessionid = s.domain_sessionid

WHEN NOT MATCHED THEN

INSERT

(

    app_id,

    domain_sessionid,

    domain_sessionidx,

    start_tstamp,

    end_tstamp,

    user_id,

    domain_userid,

    network_userid,

    page_views,

    engaged_time_in_s,

    absolute_time_in_s,

    first_page_title,

    first_page_url,

    first_page_urlscheme,

    first_page_urlhost,

    first_page_urlpath,

    first_page_urlquery,

    first_page_urlfragment,

    last_page_title,

    last_page_url,

    last_page_urlscheme,

    last_page_urlhost,

    last_page_urlpath,

    last_page_urlquery,

    last_page_urlfragment,

    referrer,

    refr_urlscheme,

    refr_urlhost,

    refr_urlpath,

    refr_urlquery,

    refr_urlfragment,

    refr_medium,

    refr_source,

    refr_term,

    mkt_medium,

    mkt_source,

    mkt_term,

    mkt_content,

    mkt_campaign,

    mkt_clickid,

    mkt_network,

    geo_country,

    geo_region,

    geo_region_name,

    geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_timezone,

    user_ipaddress,

    useragent,

    br_renderengine,

    br_lang,

    os_timezone,

    category,

    primary_impact,

    reason,

    spider_or_robot,

    useragent_family,

    useragent_major,

    useragent_minor,

    useragent_patch,

    useragent_version,

    os_family,

    os_major,

    os_minor,

    os_patch,

    os_patch_minor,

    os_version,

    device_family,

    device_class,

    agent_class,

    agent_name,

    agent_name_version,

    agent_name_version_major,

    agent_version,

    agent_version_major,

    device_brand,

    device_name,

    device_version,

    layout_engine_class,

    layout_engine_name,

    layout_engine_name_version,

    layout_engine_name_version_major,

    layout_engine_version,

    layout_engine_version_major,

    operating_system_class,

    operating_system_name,

    operating_system_name_version,

    operating_system_version

)

VALUES

(

    s.app_id,

    s.domain_sessionid,

    s.domain_sessionidx,

    s.start_tstamp,

    s.end_tstamp,

    s.user_id,

    s.domain_userid,

    s.network_userid,

    s.page_views,

    s.engaged_time_in_s,

    s.absolute_time_in_s,

    s.first_page_title,

    s.first_page_url,

    s.first_page_urlscheme,

    s.first_page_urlhost,

    s.first_page_urlpath,

    s.first_page_urlquery,

    s.first_page_urlfragment,

    s.last_page_title,

    s.last_page_url,

    s.last_page_urlscheme,

    s.last_page_urlhost,

    s.last_page_urlpath,

    s.last_page_urlquery,

    s.last_page_urlfragment,

    s.referrer,

    s.refr_urlscheme,

    s.refr_urlhost,

    s.refr_urlpath,

    s.refr_urlquery,

    s.refr_urlfragment,

    s.refr_medium,

    s.refr_source,

    s.refr_term,

    s.mkt_medium,

    s.mkt_source,

    s.mkt_term,

    s.mkt_content,

    s.mkt_campaign,

    s.mkt_clickid,

    s.mkt_network,

    s.geo_country,

    s.geo_region,

    s.geo_region_name,

    s.geo_city,

    s.geo_zipcode,

    s.geo_latitude,

    s.geo_longitude,

    s.geo_timezone,

    s.user_ipaddress,

    s.useragent,

    s.br_renderengine,

    s.br_lang,

    s.os_timezone,

    s.category,

    s.primary_impact,

    s.reason,

    s.spider_or_robot,

    s.useragent_family,

    s.useragent_major,

    s.useragent_minor,

    s.useragent_patch,

    s.useragent_version,

    s.os_family,

    s.os_major,

    s.os_minor,

    s.os_patch,

    s.os_patch_minor,

    s.os_version,

    s.device_family,

    s.device_class,

    s.agent_class,

    s.agent_name,

    s.agent_name_version,

    s.agent_name_version_major,

    s.agent_version,

    s.agent_version_major,

    s.device_brand,

    s.device_name,

    s.device_version,

    s.layout_engine_class,

    s.layout_engine_name,

    s.layout_engine_name_version,

    s.layout_engine_name_version_major,

    s.layout_engine_version,

    s.layout_engine_version_major,

    s.operating_system_class,

    s.operating_system_name,

    s.operating_system_name_version,

    s.operating_system_version

);
```

#### Users

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_web_users t

USING <SQL_RUNNER_DERIVED_SCHEMA>.users s

ON T.domain_userid = s.domain_userid

WHEN NOT MATCHED THEN

INSERT

(

    user_id,

    domain_userid,

    network_userid,

    start_tstamp,

    end_tstamp,

    page_views,

    sessions,

    engaged_time_in_s,

    first_page_title,

    first_page_url,

    first_page_urlscheme,

    first_page_urlhost,

    first_page_urlpath,

    first_page_urlquery,

    first_page_urlfragment,

    last_page_title,

    last_page_url,

    last_page_urlscheme,

    last_page_urlhost,

    last_page_urlpath,

    last_page_urlquery,

    last_page_urlfragment,

    referrer,

    refr_urlscheme,

    refr_urlhost,

    refr_urlpath,

    refr_urlquery,

    refr_urlfragment,

    refr_medium,

    refr_source,

    refr_term,

    mkt_medium,

    mkt_source,

    mkt_term,

    mkt_content,

    mkt_campaign,

    mkt_clickid,

    mkt_network

)

VALUES

(

    s.user_id,

    s.domain_userid,

    s.network_userid,

    s.start_tstamp,

    s.end_tstamp,

    s.page_views,

    s.sessions,

    s.engaged_time_in_s,

    s.first_page_title,

    s.first_page_url,

    s.first_page_urlscheme,

    s.first_page_urlhost,

    s.first_page_urlpath,

    s.first_page_urlquery,

    s.first_page_urlfragment,

    s.last_page_title,

    s.last_page_url,

    s.last_page_urlscheme,

    s.last_page_urlhost,

    s.last_page_urlpath,

    s.last_page_urlquery,

    s.last_page_urlfragment,

    s.referrer,

    s.refr_urlscheme,

    s.refr_urlhost,

    s.refr_urlpath,

    s.refr_urlquery,

    s.refr_urlfragment,

    s.refr_medium,

    s.refr_source,

    s.refr_term,

    s.mkt_medium,

    s.mkt_source,

    s.mkt_term,

    s.mkt_content,

    s.mkt_campaign,

    s.mkt_clickid,

    s.mkt_network,

);
```

### Merge your existing data into the new tables (mobile)

The following SQL will merge the existing mobile records in your SQL Runner derived tables into the new dbt derived tables, please run each one as required. This approach will leave any columns that only exist in dbt `null`. Please ensure you replace the schema and table names with your ones where appropriate.

**If you are using Redshift, be sure to `commit` your changes.**

> **Note:** It is possible, particularly for columns which may have been null, that the types of columns across the two tables don't entirely match. Your warehouse may manage this for you, or you may have to use a `cast(col_name as new_type)` in place of just selecting the column based on any error message you receive.

#### Screen Views

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_screen_views t

USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_screen_views s

ON T.screen_view_id = s.screen_view_id

WHEN NOT MATCHED THEN

INSERT

(

    screen_view_id,

    event_id,

    app_id,

    user_id,

    device_user_id,

    network_userid,

    session_id,

    session_index,

    previous_session_id,

    session_first_event_id,

    screen_view_in_session_index,

    screen_views_in_session,

    dvce_created_tstamp,

    collector_tstamp,

    derived_tstamp,

    model_tstamp,

    screen_view_name,

    screen_view_transition_type,

    screen_view_type,

    screen_fragment,

    screen_top_view_controller,

    screen_view_controller,

    screen_view_previous_id,

    screen_view_previous_name,

    screen_view_previous_type,

    platform,

    dvce_screenwidth,

    dvce_screenheight,

    device_manufacturer,

    device_model,

    os_type,

    os_version,

    android_idfa,

    apple_idfa,

    apple_idfv,

    open_idfa,

    device_latitude,

    device_longitude,

    device_latitude_longitude_accuracy,

    device_altitude,

    device_altitude_accuracy,

    device_bearing,

    device_speed,

    geo_country,

    geo_region,

    geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_region_name,

    geo_timezone,

    user_ipaddress,

    useragent,

    carrier,

    network_technology,

    network_type,

    build,

    version

)

VALUES

(

    s.screen_view_id,

    s.event_id,

    s.app_id,

    s.user_id,

    s.device_user_id,

    s.network_userid,

    s.session_id,

    s.session_index,

    s.previous_session_id,

    s.session_first_event_id,

    s.screen_view_in_session_index,

    s.screen_views_in_session,

    s.dvce_created_tstamp,

    s.collector_tstamp,

    s.derived_tstamp,

    s.model_tstamp,

    s.screen_view_name,

    s.screen_view_transition_type,

    s.screen_view_type,

    s.screen_fragment,

    s.screen_top_view_controller,

    s.screen_view_controller,

    s.screen_view_previous_id,

    s.screen_view_previous_name,

    s.screen_view_previous_type,

    s.platform,

    s.dvce_screenwidth,

    s.dvce_screenheight,

    s.device_manufacturer,

    s.device_model,

    s.os_type,

    s.os_version,

    s.android_idfa,

    s.apple_idfa,

    s.apple_idfv,

    s.open_idfa,

    s.device_latitude,

    s.device_longitude,

    s.device_latitude_longitude_accuracy,

    s.device_altitude,

    s.device_altitude_accuracy,

    s.device_bearing,

    s.device_speed,

    s.geo_country,

    s.geo_region,

    s.geo_city,

    s.geo_zipcode,

    s.geo_latitude,

    s.geo_longitude,

    s.geo_region_name,

    s.geo_timezone,

    s.user_ipaddress,

    s.useragent,

    s.carrier,

    s.network_technology,

    s.network_type,

    s.build,

    s.version,

);
```

#### Sessions

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_sessions t

USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_session s

ON T.session_id = s.session_id

WHEN NOT MATCHED THEN

INSERT

(

    app_id,

    session_id,

    session_index,

    previous_session_id,

    session_first_event_id,

    session_last_event_id,

    start_tstamp,

    end_tstamp,

    model_tstamp,

    user_id,

    device_user_id,

    network_userid,

    session_duration_s,

    has_install,

    screen_views,

    screen_names_viewed,

    app_errors,

    fatal_app_errors,

    first_event_name,

    last_event_name,

    first_screen_view_name,

    first_screen_view_transition_type,

    first_screen_view_type,

    last_screen_view_name,

    last_screen_view_transition_type,

    last_screen_view_type,

    platform,

    dvce_screenwidth,

    dvce_screenheight,

    device_manufacturer,

    device_model,

    os_type,

    os_version,

    android_idfa,

    apple_idfa,

    apple_idfv,

    open_idfa,

    device_latitude,

    device_longitude,

    device_latitude_longitude_accuracy,

    device_altitude,

    device_altitude_accuracy,

    device_bearing,

    device_speed,

    geo_country,

    geo_region,

    geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_region_name,

    geo_timezone,

    user_ipaddress,

    useragent,

    name_tracker,

    v_tracker,

    carrier,

    network_technology,

    network_type,

    first_build,

    last_build,

    first_version,

    last_version

)

VALUES

(

    s.app_id,

    s.session_id,

    s.session_index,

    s.previous_session_id,

    s.session_first_event_id,

    s.session_last_event_id,

    s.start_tstamp,

    s.end_tstamp,

    s.model_tstamp,

    s.user_id,

    s.device_user_id,

    s.network_userid,

    s.session_duration_s,

    s.has_install,

    s.screen_views,

    s.screen_names_viewed,

    s.app_errors,

    s.fatal_app_errors,

    s.first_event_name,

    s.last_event_name,

    s.first_screen_view_name,

    s.first_screen_view_transition_type,

    s.first_screen_view_type,

    s.last_screen_view_name,

    s.last_screen_view_transition_type,

    s.last_screen_view_type,

    s.platform,

    s.dvce_screenwidth,

    s.dvce_screenheight,

    s.device_manufacturer,

    s.device_model,

    s.os_type,

    s.os_version,

    s.android_idfa,

    s.apple_idfa,

    s.apple_idfv,

    s.open_idfa,

    s.device_latitude,

    s.device_longitude,

    s.device_latitude_longitude_accuracy,

    s.device_altitude,

    s.device_altitude_accuracy,

    s.device_bearing,

    s.device_speed,

    s.geo_country,

    s.geo_region,

    s.geo_city,

    s.geo_zipcode,

    s.geo_latitude,

    s.geo_longitude,

    s.geo_region_name,

    s.geo_timezone,

    s.user_ipaddress,

    s.useragent,

    s.name_tracker,

    s.v_tracker,

    s.carrier,

    s.network_technology,

    s.network_type,

    s.first_build,

    s.last_build,

    s.first_version,

    s.last_version

);
```

#### Users

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_users t

USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_users s

ON t.device_user_id = s.device_user_id

WHEN NOT MATCHED THEN

INSERT

(

    user_id,

    device_user_id,

    network_userid,

    start_tstamp,

    end_tstamp,

    model_tstamp,

    screen_views,

    screen_names_viewed,

    sessions,

    sessions_duration_s,

    active_days,

    app_errors,

    fatal_app_errors,

    first_screen_view_name,

    first_screen_view_transition_type,

    first_screen_view_type,

    last_screen_view_name,

    last_screen_view_transition_type,

    last_screen_view_type,

    platform,

    dvce_screenwidth,

    dvce_screenheight,

    device_manufacturer,

    device_model,

    os_type,

    first_os_version,

    last_os_version,

    android_idfa,

    apple_idfa,

    apple_idfv,

    open_idfa,

    geo_country,

    geo_region,

    geo_city,

    geo_zipcode,

    geo_latitude,

    geo_longitude,

    geo_region_name,

    geo_timezone,

    first_carrier,

    last_carrier

)

VALUES

(

    s.user_id,

    s.device_user_id,

    s.network_userid,

    s.start_tstamp,

    s.end_tstamp,

    s.model_tstamp,

    s.screen_views,

    s.screen_names_viewed,

    s.sessions,

    s.sessions_duration_s,

    s.active_days,

    s.app_errors,

    s.fatal_app_errors,

    s.first_screen_view_name,

    s.first_screen_view_transition_type,

    s.first_screen_view_type,

    s.last_screen_view_name,

    s.last_screen_view_transition_type,

    s.last_screen_view_type,

    s.platform,

    s.dvce_screenwidth,

    s.dvce_screenheight,

    s.device_manufacturer,

    s.device_model,

    s.os_type,

    s.first_os_version,

    s.last_os_version,

    s.android_idfa,

    s.apple_idfa,

    s.apple_idfv,

    s.open_idfa,

    s.geo_country,

    s.geo_region,

    s.geo_city,

    s.geo_zipcode,

    s.geo_latitude,

    s.geo_longitude,

    s.geo_region_name,

    s.geo_timezone,

    s.first_carrier,

    s.last_carrier

);
```

#### App Errors

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_app_errors t

USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_app_errors s

ON t.event_id = s.event_id

WHEN NOT MATCHED THEN

INSERT

(

    event_id,

        app_id,

        user_id,

        device_user_id,

        network_userid,

        session_id,

        session_index,

        previous_session_id,

        session_first_event_id,

        dvce_created_tstamp,

        collector_tstamp,

        derived_tstamp,

        model_tstamp,

        platform,

        dvce_screenwidth,

        dvce_screenheight,

        device_manufacturer,

        device_model,

        os_type,

        os_version,

        android_idfa,

        apple_idfa,

        apple_idfv,

        open_idfa,

        screen_id,

        screen_name,

        screen_activity,

        screen_fragment,

        screen_top_view_controller,

        screen_type,

        screen_view_controller,

        device_latitude,

        device_longitude,

        device_latitude_longitude_accuracy,

        device_altitude,

        device_altitude_accuracy,

        device_bearing,

        device_speed,

        geo_country,

        geo_region,

        geo_city,

        geo_zipcode,

        geo_latitude,

        geo_longitude,

        geo_region_name,

        geo_timezone,

        user_ipaddress,

        useragent,

        carrier,

        network_technology,

        network_type,

        build,

        version,

        event_index_in_session,

        message,

        programming_language,

        class_name,

        exception_name,

        is_fatal,

        line_number,

        stack_trace,

        thread_id,

        thread_name

)

VALUES

(

    s.event_id,

    s.app_id,

    s.user_id,

    s.device_user_id,

    s.network_userid,

    s.session_id,

    s.session_index,

    s.previous_session_id,

    s.session_first_event_id,

    s.dvce_created_tstamp,

    s.collector_tstamp,

    s.derived_tstamp,

    s.model_tstamp,

    s.platform,

    s.dvce_screenwidth,

    s.dvce_screenheight,

    s.device_manufacturer,

    s.device_model,

    s.os_type,

    s.os_version,

    s.android_idfa,

    s.apple_idfa,

    s.apple_idfv,

    s.open_idfa,

    s.screen_id,

    s.screen_name,

    s.screen_activity,

    s.screen_fragment,

    s.screen_top_view_controller,

    s.screen_type,

    s.screen_view_controller,

    s.device_latitude,

    s.device_longitude,

    s.device_latitude_longitude_accuracy,

    s.device_altitude,

    s.device_altitude_accuracy,

    s.device_bearing,

    s.device_speed,

    s.geo_country,

    s.geo_region,

    s.geo_city,

    s.geo_zipcode,

    s.geo_latitude,

    s.geo_longitude,

    s.geo_region_name,

    s.geo_timezone,

    s.user_ipaddress,

    s.useragent,

    s.carrier,

    s.network_technology,

    s.network_type,

    s.build,

    s.version,

    s.event_index_in_session,

    s.message,

    s.programming_language,

    s.class_name,

    s.exception_name,

    s.is_fatal,

    s.line_number,

    s.stack_trace,

    s.thread_id,

    s.thread_name

);
```

---

# Mobile data model for SQL Runner
> Legacy SQL Runner mobile data model for transforming mobile event data into derived tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-mobile-data-model/

> **Tip:** For any new developments we highly recommend using our [unified digital package for dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) instead. SQL Runner is no longer under active development and will only receive bug fixes in the future.

## Overview

The Snowplow mobile data model aggregates Snowplow's out-of-the-box mobile events to create a set of derived tables - screen views, sessions, and users. These contain many useful dimensions, as well as calculated measures such as screen views per session.

![Diagram showing the three levels of the SQL Runner mobile data model: ENTITY (users) at the top, CYCLE (sessions) in the middle, and INTERACTION (screen views) at the bottom, connected by vertical lines.](/assets/images/Screenshot-2022-03-25-at-10.50.33-5d404dca4e545db52a48114f0750bd14.png)

The Snowplow mobile data model processes and aggregates data collected by the Snowplow [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) and [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) mobile trackers into a set of derived tables - screen views, sessions and users - with each table generated by its respective module.

In addition to these 'standard' modules there are optional modules that can be enabled depending on your tracker implementation. Currently there is only one such optional module, Application Errors, which provides application crash reporting. These modules produce their own derived output table, as well as being aggregated and joined into 'standard' modules such as sessions for high level KPIs. As the tracker's functionality continues to expand, so will this library of optional modules.

Unlike the [web model](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/) the level of aggregation is not linear, as shown in the diagram below. This allows the sessions module to consume events other than just screen views, such as app installs or crashes.

![Comparison diagram showing the difference in aggregation structure between the Mobile model (top) and Web model (bottom). In the Mobile model, Atomic Events feeds into both Optional Modules and Screen Views in parallel, and both feed into Sessions, then Users. In the Web model, Atomic Events feeds linearly into Page Views, then Sessions, then Users.](/assets/images/web_vs_mobile-c178efd12abd487001ed45f836868293.jpg)

The model runs incrementally, only processing new events (and events that have previously been modeled but are part of screen views, sessions or users for which/whom there is new information) with every run. The incremental logic is separate from the logic that calculates the tables so as to make customization of the model easier. More information on the model structure and customization options can be found below.

**The latest Snowplow mobile data model can be found in the [snowplow/data-models GitHub repository](https://github.com/snowplow/data-models/tree/master).**

#### Supported Warehouses

- Redshift
- BigQuery
- Snowflake

#### Requirements

- Snowplow [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) or [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) mobile tracker version 1.1.0 or later implemented.
- Mobile session context enabled.
- Screen view events enabled.

## Quick start

#### Prerequisites

- [SQL-runner](https://github.com/snowplow/sql-runner) is installed.
- Snowplow [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) or [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) mobile tracker implemented with mobile events dataset present in a database.

#### Playbook Configuration

##### Setting Schemas

For all playbooks within `mobile/v1/{warehouse}/sql-runner/playbooks/standard`, configure the following:

- `input_schema` - Set to the schema containing your Snowplow events data.
- `scratch_schema` - Set to the scratch/staging schema to contain the intermediate tables generated by this model. Create if required.
- `output_schema` - Set to a schema to contain the model's final output tables (`mobile_screen_views`, `mobile_sessions` etc.)

##### Base Module

Within the base module's main playbook, `01-base-main.yml.tmpl`:

1. Set the `start_date` as desired.

2. Enable the following contexts as desired by setting to `True`:

   - Mobile - Device type, OS etc.
   - Geolocation - Device latitude, longitude, bearing etc.
   - Application - App version and build.
   - Screen - Screen details associated with a mobile event.

3. Adjust filtering of events if required:

   - `platform_filters`: Default - `platform = 'mob'`. Override if required.
   - `app_id`: Default - No filter on `app_id`. Add if required.

```yaml
# 01-base-main.yml.tmpl

:variables:

   ...

   :start_date:         2020-01-01 #Set as required

   ...

   #Enable contexts if desired

   :mobile_context: false

   :geolocation_context: false

   :application_context: false

   :screen_context: false

   :platform_filters: [] #Override default if required e.g. ['mob','tv']

   :app_id_filters: [] #Add app_id filter if required e.g ['my_app']
```

The remaining variables are set to the recommended defaults. For more information please see the base module's README within the playbooks directory.

##### Optional Modules

Optional modules are disabled by default. Please enable where appropriate by setting `enabled: true` in the main playbook of each module.

#### Running the model

##### Authentication

1. Set database password as an environmental variable.

   - Redshift: Set environmental variable, `REDSHIFT_PASSWORD`, to your database password.
   - BigQuery: Set environmental variable, `GOOGLE_APPLICATION_CREDENTIALS`, to the path of your GBQ json credential file
   - Snowflake: Set environmental variable, `SNOWFLAKE_PASSWORD`, to your database password.

2. Fill in the relevant template in `.scripts/templates` with your database details.
   - Redshift & Snowflake: Leave `PASSWORD_PLACEHOLDER` as is. This placeholder will be replaced at run time with the credentials from env var `{WAREHOUSE}_PASSWORD`, set in step 1.

##### Execution

For a full run of the model:

```bash
bash .scripts/run_config.sh -b ~/pathTo/sql-runner -c mobile/v1/{warehouse}/sql-runner/configs/datamodeling.json -t .scripts/templates/{warehouse}.yml.tmpl;
```

For more details please see the README within the `.scripts` directory.

##### Authentication

For each playbook within `mobile/v1/{warehouse}/sql-runner/playbooks/standard` fill in your database connection details. For more details please refer to the [SQL Runner Docs](/docs/modeling-your-data/modeling-your-data-with-sql-runner/#target-configuration).

##### Execution

There are many tools available to schedule and run SQL jobs such as the mobile model. While these tools differ in operation the basic principles to running the mobile model should be similar.

A series of tasks will need to be set to execute SQL Runner against each individual playbook of the model in turn. For example, to run the main playbook of the base module:

```bash
bash {sql-runner-path} -playbook sql-runner/playbooks/standard/01-base/01-base-main.yml.tmpl -sqlroot sql-runner/sql
```

The correct order of playbook execution and the dependancies between them can be determined from the datamodeling config file within the repo.

#### Backfilling

By default when first running the model, events will be processed in 7 day increments beginning at the `start_date` set in the base module's playbook, `01-base-main.yml.tmpl`, as described above. To backfill data faster, set the `update_cadence_days` in the `01-base-main.yml.tmpl` to required number of days. Once backfilled, it is recommended to set `update_cadence_days` back to 7 days for performance.

#### Testing

The mobile model comes with a suite of data validation checks to ensure the model is running as intended. These checks are performed by [Great Expectations](https://greatexpectations.io/). The tests were used during the model's development and are not required to run the mobile model. If desired, they can however be used during the deployment of the mobile model on your own Snowplow dataset to validate the output. For further details please refer to the README in the `.scripts` directory.

#### Advanced Configurations

This quick start guide covers the implementation process for the majority of use cases. Further configurations can be made such as:

- Running specific modules more frequently than others.
- Running specific modules only.
- Adjusting table scan limits.

For further details please refer to the READMEs within each module's playbook directory.

## Technical architecture

#### The entire model

This model consists of a series of modules, each is idempotent and can be run on independent schedules, and each produces a table which serves as the input to the next module.

![Full flowchart of the SQL Runner mobile data model. From Atomic Tables/Manifests, the base module produces mobile\_events\_staged, which feeds in parallel into the screen views module (producing mobile\_screen\_views and mobile\_screen\_views\_staged) and the app errors optional module (producing mobile\_app\_errors and mobile\_app\_errors\_staged). Both staged tables feed into the sessions module (producing mobile\_sessions and mobile\_sessions\_userid\_manifest\_staged), which in turn feeds the users module (producing mobile\_users). Each module has a main playbook and a complete playbook.](/assets/images/mobile_full_model_structure-ee33577d65c009462e5a236bf9a6e92f.jpg)

#### The individual modules

The ‘standard’ modules can be thought of as source code for the core logic of the model, which Snowplow maintains. These modules carry out the incremental logic in such a way as custom modules can be written to plug into the model’s structure, without needing to write a parallel incremental logic. We recommend that all customisations are written in this way, which allows us to safely maintain and roll out updates to the model, without impact on dependent custom SQL. For more information on this, jump to the section on customizing the model below.

Each module produces a table which acts as the input to the subsequent module (the `_staged` tables), and updates a production table – with the exception of the base module, which takes atomic data as its input, and does not update a production table.

Each module comes with a `99-{module}-complete playbook`, which marks that module complete by updating any relevant manifests, and truncating the input, and cleaning up temporary tables. The complete steps may be run at the end of each module, or at the end of the run.

More detail on each module can be found in the relevant READMEs in the [GitHub repository](https://github.com/snowplow/data-models/tree/master).

![Diagram showing the structure of a single SQL Runner module: Step 1 (01-main playbook) reads from an Inputs table and outputs to Production Tables and Staged Tables; Step 2 (optional custom SQL plugin) also reads from Inputs and outputs to Custom Tables; Step 3 (99-complete playbook) truncates the Inputs table to finish the incremental lifecycle.](/assets/images/web_model_module-1-630f2fc2a9282fd3089910576d1c4c0b.jpg)

Architecture of an individual mobile module

## Customizing the model

Custom modules can fit into the incremental structure by consuming the same inputs, and running before the `99-{module}-complete` playbook runs. Custom modules may also consume and intermediary tables of the standard module, which will not be dropped until the `99-{module}-complete` playbook runs.

Any custom SQL that depends on a `_staged` table as its input should run before the complete step of the module which handles that same input. For example, custom logic which takes mobile\_events\_staged as an input should run before the `99-sessions-complete` playbook which truncates mobile\_events\_staged.

An example custom module has been included in `mobile/v1/redshift/sql-runner/sql/custom`. In this module we:

1. Read screen views from `scratch.mobile_screen_views_staged`.
2. Aggregate screen views to one row per `session_id`.
3. Delete and insert into the output table `derived.session_goals`. This table can then be joined onto `derived.mobile_sessions` on `session_id`.

The playbooks should then be run in the following order:

```text
standard/01-base/01-base-main

standard/01-base/99-base-complete

standard/02-screen-views/01-screen-views-main

standard/02-screen-views/99-screen-views-complete

standard/03-optional-modules/01-app-errors/01-app-errors-main

standard/03-optional-modules/01-app-errors/99-app-errors-complete

standard/04-sessions/01-sessions-main

custom/04-session-goals/01-session-goals-main

custom/04-session-goals/01-session-goals-complete

standard/04-sessions/01-sessions-complete
```

An example of this can be seen in the `datamodeling_custom_module.json` file within the config directory.

Custom modules can also be created with their own independent manifesting logic, in which case they are more complex, but don’t rely on the standard modules.

More information on customizing the Snowplow mobile data model can be found in the custom module [README](https://github.com/snowplow/data-models/blob/master/mobile/v1/redshift/sql-runner/sql/custom/README.md).

## Metadata

Metadata is logged in the `{{.output_schema}}.datamodel_metadata{{.entropy}}` table, per-module and per-run. A persistent ID is created, so that separate modules within the same run may be identified.

---

# Web data model for SQL Runner
> Legacy SQL Runner web data model for transforming web event data into derived tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/

> **Tip:** For any new developments we highly recommend using our [unified digital package for dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) instead. SQL Runner is no longer under active development and will only receive bug fixes in the future.

The Snowplow web data model aggregates Snowplow's out of the box page view and page ping events to create a set of derived tables - page views, sessions and users - that contain many useful dimensions as well as calculated measures such as time engaged and scroll depth.

![Diagram showing the three levels of the SQL Runner web data model: ENTITY (users) at the top, CYCLE (sessions) in the middle, and INTERACTION (page views) at the bottom, connected by vertical lines.](/assets/images/image-3-2f58ac3c0dcb0a7256a0dbd728280cef.png)

**The latest Snowplow web data model can be found in the [snowplow/data-models GitHub repository](https://github.com/snowplow/data-models/tree/master).**

## Quickstart

#### Requirements

- [Snowplow Javascript tracker](/docs/sources/web-trackers/) version 2 or later implemented.
- Web Page context [enabled](/docs/sources/web-trackers/tracker-setup/initialization-options/) (enabled by default in [v3+](/docs/sources/web-trackers/tracker-setup/initialization-options/)).
- [Page view events](/docs/sources/web-trackers/tracking-events/page-views/) implemented.

#### Prerequisites

- [SQL-runner](https://github.com/snowplow/sql-runner) must be installed ([Setup guide](/docs/modeling-your-data/modeling-your-data-with-sql-runner/)).
- A dataset of web events from the [Snowplow Javascript tracker](/docs/sources/web-trackers/) must be available in the database.

#### Configuring and running the model

First, fill in the connection details for the target database in the relevant template in `.scripts/templates/redshift.yml.tmpl`.

Password can be left as a `PASSWORD_PLACEHOLDER`, and set as an environment variable or passed as an argument to the run\_playbooks script. See the README in `.scripts` for more detail.

Variables in each module's playbook can also optionally be configured also. See each playbook directory's README for more detail on configuration of each module.

You can then run the model, either by running playbooks individually by running SQL-runner locally, or via your Snowplow GitHub repository. Of course, as a Snowplow customer you can also reach out to Support to get the model deployed for you.

## Technical architecture

#### The entire model

This model consists of a series of modules, each is idempotent and can be run on independent schedules, and each produces a table which serves as the input to the next module.

![Full flowchart of the SQL Runner web data model showing the complete module sequence from Atomic Tables/Manifests through base, page views, sessions, and users modules. Each level shows a main playbook, optional custom aggregation steps, a complete playbook, a production table output, and a staged table that feeds into the next module.](/assets/images/web_full_model_structure-317e3b071cb592772020d3073ae4cb04.jpg)

#### The individual modules

The 'standard' modules can be thought of as source code for the core logic of the model, which Snowplow maintains. These modules carry out the incremental logic in such a way as custom modules can be written to plug into the model's structure, without needing to write a parallel incremental logic. We recommend that all customisations are written in this way, which allows us to safely maintain and roll out updates to the model, without impact on dependent custom SQL. For more information on this, jump to the section on customizing the model below.

Each module produces a table which acts as the input to the subsequent module (the `_staged` tables), and updates a production table - with the exception of the base module, which takes atomic data as its input, and does not update a production table.

Each module comes with a `99-{module}-complete playbook`, which marks that module complete by updating any relevant manifests, and truncating the input, and cleaning up temporary tables. The complete steps may be run at the end of each module, or at the end of the run.

![Diagram showing the structure of a single SQL Runner module: Step 1 (01-main playbook) reads from an Inputs table and outputs to Production Tables and Staged Tables; Step 2 (optional custom SQL plugin) also reads from Inputs and outputs to Custom Tables; Step 3 (99-complete playbook) truncates the Inputs table to finish the incremental lifecycle.](/assets/images/web_model_module-630f2fc2a9282fd3089910576d1c4c0b.jpg)

More detail on each module can be found in the relevant READMEs in the [GitHub repository](https://github.com/snowplow/data-models/tree/master).

## Customizing the model

Custom modules can fit into the incremental structure by consuming the same inputs, and running before the `99-{module}-complete` playbook runs. Custom modules may also consume and intermediary tables of the standard module, which will not be dropped until the `99-{module}-complete` playbook runs.

Any custom SQL that depends on a `_staged` table as its input should run before the complete step of the module which handles that same input. For example, custom logic which takes events\_staged as an input should run before the `99-page-views-complete` playbook.

As an introductory example, if there is a requirement to include data from custom events and entities for page views, for example, we would write a custom module which:

- Reads events (i.e. event\_ids) from the `scratch.events_staged` table
- Aggregates to one row per `page_view_id`
- Deletes and inserts this to a custom table which can join to the `derived.page_views table` on `page_view_id`

If the playbook for this custom module is called `AA-my-custom-page-views-level-module.yml.tmpl`, then we would run the playbooks as follows:

- `standard/01-base/01-base-main.yml.tmpl`
- `standard/01-base/99-base-complete.yml.tmpl`
- `standard/02-page-views/01-page-views-main.yml.tmpl`
- `custom/AA-my-custom-page-views-level-module.yml.tmpl`
- `standard/02-page-views/99-page-views-complete.yml.tmpl`

Custom modules can also be created with their own independent manifesting logic, in which case they are more complex, but don't rely on the standard modules.

More information on customizing the Snowplow web data model can be found in the [`web/v1/redshift/sql-runner/sql/custom` folder on GitHub](https://github.com/snowplow/data-models/).

## Metadata

Metadata is logged in the `{{.output_schema}}.datamodel_metadata{{.entropy}}` table, per-module and per-run. A persistent ID is created, so that separate modules within the same run may be identified. This ID is automatically handled, as long as the `99-{module}-complete` step of the last module to run has the `:ends_run:` variable set to `false`.

Rather than setting this variable, this can be maintained manually be running the `00-setup/00-setup-metadata.yml.tmpl` before all other steps, and the `00-setup/99-complete-metadata.yml.tmpl` playbook after all other steps.

---

# Run data models from Console
> The guide to running data models in Snowplow.
> Source: https://docs.snowplow.io/docs/modeling-your-data/running-data-models-via-console/

You can use [Snowplow Console](https://console.snowplowanalytics.com) to schedule and run [dbt](https://www.getdbt.com/) data models, including our [out-of-the-box models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/), [automatically generated models](/docs/modeling-your-data/automatically-generated-data-models/), and any custom or third party models.

## Overview

```mermaid
flowchart LR
  subgraph console [Snowplow Console]
    schedule1(Schedule 1)
    schedule2(Schedule 2)
    schedule3(...)
  end
  subgraph "Your cloud account"
    job(Modeling job)
  end
  subgraph "Your Git repository"
    project(dbt project)
  end
  warehouse[(Warehouse)]
  console(Snowplow Console) -->|Trigger| job -->|Execute models| warehouse
  job -.->|Read configuration| project
```

To use this feature, you will need to store your data models in a **Git repository** organized into one or more [**dbt projects**](https://docs.getdbt.com/docs/build/projects). This provides you full control over the model logic and helps with versioning and collaboration.

In Console, you can define multiple **run configurations** and **schedules** to control what dbt commands get executed and when. For example, you could set the Unified Digital data model to run every day at midnight.

Based on what you defined, Console will trigger data modeling **jobs** in your cloud account associated with Snowplow (managed by you, in the case of Private Managed Cloud, or by Snowplow otherwise). The jobs will execute actual dbt commands against your data warehouse.

> **Tip:** This architecture allows you to take advantage of a secure connection to the warehouse (e.g., VPC Peering or AWS Private Link) that you might have already set up with Snowplow.

## Prerequisites

To get started, you will need a Git repository (and sufficient permissions to grant read access to Snowplow).

We also highly recommend installing dbt locally so that you can quickly test your model configuration. Follow the [dbt documentation](https://docs.getdbt.com/docs/core/installation-overview) to install dbt Core as well as the relevant adapter for your destination (e.g., `dbt-snowflake`).

## Initial setup

First, create a Git connection to your repository:

- In Console, go to **Settings > Connections > Set up connection > Git repository**
- The setup flow will guide you to create and test the connection

Next, create the model project:

- Navigate to **Modeling > Model projects > Create model project**
- You will need to provide the Git connection you created in the previous step, as well as some metadata for your project
- Console will show you the dbt commands to initialize your project if you haven't done so (you can also bring a pre-existing project)

Once the project is created, you can start adding data models.

## Add data models

> **Tip:** If you already have a dbt project with a set of data models, you can skip this section. Console will work with your current dbt configuration in `dbt_project.yml` and `packages.yml`.

Because the dbt configuration is stored in your Git repository, rather than in Console, you will add and configure models by making commits to this repository.

For example, to install our Unified Digital data model, you can follow the [tutorial](/tutorials/unified-digital/intro/) and add `snowplow/snowplow_unified` to your `packages.yml`.

Likewise, to configure variables for the models, edit the `dbt_project.yml` file.

With multiple models, you might run into a situation where one depends on the output of another. As long as you use [the `ref` function](https://docs.getdbt.com/faqs/Models/create-dependencies), dbt will automatically infer and honor dependencies. In particular, see how to set up our Unified Digital and Attribution models to [work together](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/keeping-cohesion-with-unified/).

> **Note:** When running the models, Console will override any warehouse connection settings with the ones you will configure in the next section.

Once you are happy with the dbt project setup, run dbt commands locally to verify it.

## Connect to the warehouse

In this step, you will create a warehouse connection for the models to use. Note that you can define more than one connection, for example, if you would like to have separate model run configurations for production and QA data.

> **Tip:** You might have already set up a warehouse connection for loading the data. Data modeling, however, requires a new connection, as you will often want data models to run under a different user/role and with different permissions.

Navigate to **Settings > Connections > Set up connection > Data modeling**. Select your warehouse and follow the steps in Console to create and test your connection.

## Add run configurations

For Console to run your data models, you will need to create run configurations.

Each run configuration controls what dbt commands will be executed and when. More concretely:

- You can add one or more run configurations to a project
- Each run configuration can contain one or more schedules
- For each schedule, you can define specific dbt commands

> **Tip:** If you want to run all your models every day at midnight, you would create:
>
> - One run configuration
> - One schedule (daily, 12 AM)
> - Add standard dbt commands, e.g. `dbt deps` and `dbt run`
>
> If you want to run different models within the project at different times, you can use dbt’s [selector features](https://docs.getdbt.com/reference/commands/run#running-specific-models). In this case, create:
>
> - One run configuration
> - Two schedules, e.g. daily at 12 AM and every Monday at 3 AM
> - Pick different commands for each schedule, e.g. `dbt run --select <...>`
>
> In both cases, note that `dbt run` will automatically run all the relevant models in the correct order, [accounting for dependencies](https://docs.getdbt.com/faqs/Models/create-dependencies).

To create a run configuration:

- Navigate to your model project in Console
- Click **Add run configuration**
- Pick the warehouse connection created earlier
- Click **Add schedule**
- Select the desired time and dbt commands
- Repeat as necessary

## Run the models

Once you’ve defined the run configurations and schedules, Console will automatically run your data models. You can see when the next run is scheduled by looking at each run configuration.

![Run configurations panel showing an active \&quot;Main run configuration\&quot; on BigQuery with one schedule, and an \&quot;example-models\&quot; schedule set to run every 20 minutes with the next run today at 20:00](/assets/images/run-configurations-85b1a6e33da4d61b7835cbbda702f0aa.png)

You can also use the **Run now** button to trigger a one-off model run for a specific schedule. The resulting job will use the same commands and settings.

## Troubleshooting

See [Resolving data model failures](/docs/modeling-your-data/running-data-models-via-console/resolving-data-model-failures/).

---

# Resolve data model failures
> Troubleshoot and resolve data model failures in Snowplow Console including debugging steps and common issues.
> Source: https://docs.snowplow.io/docs/modeling-your-data/running-data-models-via-console/resolving-data-model-failures/

## Receive alerts

When you create a model project in Console, you have to specify one or more owner email addresses. (You can edit the project later to add or remove owners.)

Console uses these email addresses to send notifications when a data modeling job fails for any reason. The email will contain a link to the **Jobs** section in Console with the relevant failure.

## Diagnose failures

You will be able to see the details of your data model failure in the jobs interface in Console. Under **Job DAG** (Directed Acyclic Graph) for the failed job, click on the failed step to see details of what went wrong.

![Job DAG showing two steps: git-clone-cache (succeeded in 2 seconds) and run-dbt (failed in 16 seconds)](/assets/images/dbt-dag-390b9fb149c227f96a6367d15175d55b.png)

The **Error Output** will show you the error logs from the dbt commands that were executed. These logs will contain the failure information that dbt and the warehouse relayed back to Console.

![Step logs panel for run-dbt showing FAILED status after 16 seconds, with dbt output warning about deprecated configuration keys and a Database Error](/assets/images/dbt-step-error-output-2533cadbba55a302223042fe12de784e.png)

## Understand and resolve issues

An error message such as 'connection refused' or `EOF` (end of file) is typically caused by a network problem. These errors often do not need any interaction to be resolved.

If the logs show a dbt call with a model error, you will need to identify the issue in your model files and push the changes to your Git repository. For help debugging dbt errors, please see the [dbt docs](https://docs.getdbt.com/guides/debug-errors) and ensure your models run correctly locally before pushing changes.

> **Note:** Even if a job fails, it will run again next time as per the schedule (e.g. next midnight). If you identified an issue, you can temporarily disable the schedule to prevent spurious failures while you work on a fix.

---

# Retrieve data model execution data
> Retrieve data model execution data via API for monitoring and integration with alerting infrastructure.
> Source: https://docs.snowplow.io/docs/modeling-your-data/running-data-models-via-console/retrieving-job-execution-data-via-the-api/

The API that powers the warehouse jobs monitoring view in Snowplow Console (Jobs) is also available for consumption by other authenticated clients.

The exact same data about past and current jobs executions can be retrieved and processed programmatically. Hence, it possible to integrate with your monitoring infrastructure and enable additional alerting or insights.

## Getting started

You can have a look at and interact with all available endpoints in the [API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml#/Jobs).

### Authorization

To be able to post sample requests in the documentation you need to click the `Authorize` button at the top of the document and authorize with your token. The value for the token field in each individual requests is overwritten by this authorization.

The endpoints focus on the main operations in the workflow around:

1. Getting a list of job runs within a specific time window. This window **may span 96 hours at most and be contained within the last week**.
2. Retrieving information about a particular job run, including its individual steps.
3. Retrieving only the steps of a particular job run without any additional information.
4. Getting the full data available about the execution of a particular step of a particular job run, including `stdout`/`stderr`.

Each request will need to include your Organization ID. You can find it [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

Follow the instructions in the [Account management](/docs/account-management/) section to obtain an access token for API authentication.

## Retrieve job run information

### Getting a list of job runs for a specific time window

The invocation is as such:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs?from=X&to=Y`

`X` and `Y` are required and can be passed either as ISO8601 timestamps or as ISO8601 durations, and they must both be of the same type (i.e. it is not possible to mix a timestamp with a duration). For example, the query parameters can be:

- `from=2020-12-02T03:40:00Z&to=2020-12-02T15:40:00Z`
- `from=PT-12H&to=PT0S`

As mentioned earlier, the total time window requested cannot exceed 96 hours (4 days) or start earlier than 1 week ago.

Limitations to be aware of

When it comes to job executions, our platform tracks state change events. A side-effect of this is that in extreme cases of jobs with very significant numbers or steps or far too frequent executions the events for the requested time window may have to be automatically trimmed to the latest 10,000. This guarantees an up-to-date view but may be missing job runs that started near the beginning of the window. In this edge case the problem can be side-stepped by splitting the window to more requests (e.g. 2 + 2 days).

### Retrieving information about a specific job run

Can be invoked as:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs/{runId}`

`runId` is a hashed value that can be found within results of job run listing queries. The extra information that this endpoint returns (compared to job listing results) is the list of steps in that job run and their state, alongside their inter-dependencies.

### Retrieving steps of a specific job run

If it is preferred to get only the steps for a job run, without the information already contained within the job runs list, one can use the respective endpoint:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs/{runId}/steps`

### Retrieving detailed information about a specific step execution

Finally, one may wish to get the complete information about the execution of a single step. This may be useful, for instance, if the job failed on that step and the standard output or standard error of the step is of interest. To get that, one can invoke the respective endpoint:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs/{runId}/steps/{stepName}`

---

# Marketing Attribution visualization
> Marketing attribution visualization for multi-channel journeys with first-touch, last-touch, linear, and position-based models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/attribution-modeling/

In today's increasingly complex digital world, users often take multi-channel journeys before converting. Assigning credit across multiple touchpoints is vital to getting an accurate picture of the efficacy of your marketing channels, yet requires merging disparate datasets and running complex calculations.

The Snowplow Marketing Attribution visualization (together with the [Snowplow Attribution dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)) lowers the barrier to entry for your marketing team through the following features:

- Incremental SQL model in your warehouse for cost-effective computation
- Choice of first-touch, last-touch, linear and positional methods, with additional filters and transforms available
- Reports for conversions, revenue, spend and Return On Advertising Spend (ROAS) per channel and campaign
- Option to specify your own touchpoint and advertising spend tables
- Intermediate tables that you can build your own attribution models on top of

![Marketing Attribution overview dashboard showing key metrics for the last 28 days: 384 conversions, £13,268 revenue, £66,957 channel spend, and 0.198 ROAS, with bar charts showing Paid Search leading in both attributed conversions and attributed revenue by channel](/assets/images/overview-f26055aba0a5f6c9837854a24c45a0c5.png)

## Requirements

- [Campaign Attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) enabled
- [Referrer Parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) enabled
- Running the [Snowplow Unified Digital dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) with `conversion event(s)` defined and the optional conversion module enabled
- Running the [Snowplow Attribution dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)
- Access to the derived tables granted to the role used when setting up the visualization

## Configuration

Configuration is best performed by a Data Analyst or Engineer.

All these settings are global for all users, meaning if you change them they will be changed for everyone. The first user of the visualization will have to define at least one **view** which is the dataset needed to generate the charts. Creating views can be done on the **Settings** page.

### Update method

Use the toggle **Last N days View (Dynamic)** to choose whether to define a dynamic view that auto-updates or a static view:

- **Dynamic (Last N Days)**: the so-called dynamic views are for generating datasets that have a rolling conversion window of last nth day and will be refreshed automatically (e.g. last 30 days).

  The visualization will save the last-refreshed date with the view configurations. Any subsequent day a user logs back into the visualization, a query will run in the background to look for any newly processed conversion events in the conversion source. If there are any, the dynamic datasets are refreshed by running all the queries that are needed to generate data for the charts to populate.

  If you choose this option, set the **auto-update days**: the number of days since the last conversion event defined here will define the conversion window. The latest conversion window in use can be checked on the **Settings** page where a table with information on all the created views is displayed, including the conversion window that is currently in use.

- **Static (Custom Date Range)**: non-dynamic views need a name, and will typically be used to generate a fixed dataset (e.g. Jan, Q1, 2023) to avoid having to recalculate the analysis for subsequent users.

  Define a fixed conversion window by selecting the appropriate date range with the date picker tool. It is activated by clicking on the default date range.

### Set a currency symbol

The currency symbol defaults to "$".

### Connect to data sources

1. Select your schema that contains the derived unified and attribution tables. This will trigger an update which checks for any tables with the names closest to what the visualization expects.

2. After waiting for the update to take place, you can check if the auto-detected source tables are what you expect. Change them to any other appropriate existing tables if they are not correct.

3. For most users, the **Paths To Non-Conversion** table should be set to **Do not use paths\_to\_non\_conversion table**.

   This drop and recompute table calculates the paths your customers have followed that have not led to a conversion, for use in the **Path Summary** page. This table is not recalculated by the visualization. Therefore it should only be used for a static view with the intention of consuming the same period as in the latest data model.

4. Overwrite the **Attribution Manifest** table. Most likely the schema name will have to be modified. Keep the `schema_name.table_name` notation here, and press enter once modified.

5. (Optional but recommended) Specify the **Spend Source** to get the ROAS calculation in your overview. This will most likely be a view you created on top of your table that holds your marketing spend data. The view will make sure you align the expected field names. It should have `campaign`, `channel`, `spend` and `spend_tstamp` for the analysis to work. Press enter once modified.

6. Once you are happy with all the inputs, press the **Create View** button. The visualization will first run a validation against the data sources making sure it has all the fields it needs. If all is well, it will save the view.

7. The dashboards are ready to be explored. The first time a dashboard page is visited, the relevant query will run once and the data will be cached to speed up subsequent dashboard explorations for other users.

## Using the Dashboard

Once at least one view is configured, all users can go ahead and use the dashboards directly.

### Filters

There are various filters at the top of each dashboard page that make the data exploration interactive. Because the queries are cached, users can make any of these interactive changes without involving the warehouse, avoiding expensive queries or laggy information retrieval.

To use a filter, first select which view to use, and then make changes with **View Settings**:

- select which **Attribution Type** to use (`First Touch`, `Last Touch`, `Linear` or `Position Based`)
- choose between using **Campaign** or **Channel** for paths

Some pages have additional filters, such as the **Number of items** filter for the **Path Summary** page, which reduces the number of items shown in specific charts.

## Editing and deleting views

Edit or delete existing views from the **Settings** page, using the **Edit/Delete Views** dropdown.

- **Delete**: click X next to the view.
- **Edit**: click on the name of the view. The visualization will take you to the view configuration page where you can make amendments. Saving will overwrite the existing view configurations.

![Edit/Delete Views dropdown showing two saved views — \&quot;Last 30 days (Dynamic)\&quot; and \&quot;Jan\&quot; — each with a delete button](/assets/images/edit_delete_views-b9c2000c24b4add79f8cccd6b992a73b.png)

---

# Ecommerce Analytics visualization
> Ecommerce analytics visualization with dashboards for revenue, purchases, checkout journey, and transaction patterns.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/ecommerce-analytics/

The Ecommerce Analytics visualization offers an in-depth breakdown of ecommerce activities across your websites and mobile apps, presenting an enhanced rendition of Google Analytics v4 reports. Use the dashboards to:

- Access real-time data directly within your data warehouse in a governed manner.
- Navigate through a series of interactive dashboards designed to surface monetization data, purchasing behaviors, checkout experiences, and transaction patterns.
- Customize your analytics workflow with filters for campaigns, channels, and products.

Leveraging Snowplow's Ecommerce event tracking packages and our Ecommerce dbt package, the visualization transforms raw event data into structured tables covering carts, checkouts, products, transactions, and user interactions, providing valuable insights for informed decision-making.

![E-commerce Analytics overview page showing sample data for January 2023 with key metrics (total revenue $30,243.69, 162 customers, average order value $156.70) and line charts for revenue per day and unique customers per day, plus purchased products tables](/assets/images/overview-3594c6db8bee3690796595f15179f51b.png)

## Requirements

- Running the [Snowplow Ecommerce dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/)
- Access to the derived tables granted to the role used when setting up the visualization
- Ecommerce events tracked using the ecommerce APIs for the [JavaScript (Snowplow Ecommerce plugin)](/docs/sources/web-trackers/tracking-events/ecommerce/) or [mobile trackers](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/)

## Usage

The following steps will guide you through using the visualization.

### Explore the dashboards using sample data (optional)

Before you configure your data, you are free to explore all the dashboards using the sample data included. The use of sample data is enabled by default and can be enabled/disabled using the **Use Sample Data** toggle on the **Settings** page.

You can browse the following pages:

1. **Overview**: key metrics including the revenue, number of customers and the average order value. It also shows the revenue for top performing products and product lists.
2. **Purchases**: dive deeper into the purchased products and compare product views, add to carts, units purchased, and more.
3. **Sessions**: a funnel of how users navigated from product views all the way to transactions by their number of sessions, conversion rate and abandonments.
4. **Checkout**: individual checkout steps within the funnel, showing conversion rate and abandonment for each step.
5. **Transactions**: key metrics for the transactions including types of payments.

### Configure using the Ecommerce dbt package tables

The **Settings** page lets you choose the schema and tables in your data warehouse to use. These refer to the output of the Ecommerce dbt package.

First, choose the warehouse schema where your dbt Ecommerce package produced the derived tables (e.g. `dbt_ecom_derived`). The visualization will suggest the most likely tables below. The following derived tables will need to be configured: `cart interactions`, `checkout interactions`, `product interactions`, `sessions`, and `transaction interactions`.

Verify that these match the desired tables and click **Save Settings**. If the tables have the required columns, you can navigate to the dashboard pages in the sidebar and explore your data.

### Choose the date range and filters

At the top of each page, there is an option to choose the date range (including a secondary date range to compare the data against) and filters. These options let you narrow down the data shown in the dashboard to what you are interested in.

![Top of the Checkout page showing a \&quot;Last 30 Days\&quot; date range dropdown and a Filters button](/assets/images/filters-6234c465859d437817a38593a0cb7aa0.png)

When you click on the **Filters** button, you will see a modal with an option to add conditions on fields in the data. For example, you can add conditions on the products or product lists that you are interested in.

![Filter dialog configured with field \&quot;step\&quot;, operator \&quot;contains\&quot;, and value \&quot;order complete\&quot;, with Add Filter and Save buttons](/assets/images/filters_modal-3da85a650b812050f1b451d9c7701cca.png)

---

# Funnel Builder visualization
> Build custom funnel analyses to understand user journeys, conversion rates, and drop-off points across any sequence of events.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/funnel-builder/

Funnels are an essential tool for understanding user journeys on your app or website. They help to visualize how many users complete each event along a journey such as signing up or making a purchase, so you can understand which stages are leading to the most drop-off and make changes to improve conversion rates.

This visualization provides an intuitive UI for building a funnel analysis and visualizing the results. You can specify any number of conditions and steps, and will receive the following outputs when you run the analysis (see screenshots below):

- User Counts by Funnel Step Chart
- Conversion Rates Chart
- Abandonment Rates Chart
- Summary Statistics Table

It works on any table that Snowplow’s Data Modeling User has access to, including the atomic events table and derived tables. It comes with some pre-built funnels based on out-of-the-box Snowplow events such as page views and link clicks, and you can save your own custom funnels to share with teammates.



## Requirements

- Access to the table(s) you wish to run the tool on, granted to the role used when setting up the visualization

## Building a funnel

The following steps will guide you through building a funnel.

### Start from template or create a new funnel

The main page shows a list of example funnels for common use cases (e.g., ecommerce) to get you started.

You can choose to start using one of the templates, or create a new funnel configuration from scratch.

### Choose your funnel name, database schema and table

A page with the funnel configuration will be shown.

First, you are asked to provide a name for the funnel, which will be used to identify it.

Next, you can select which warehouse schema and table you want to run the analysis on. If you select the atomic events table, we have included some additional functionality allowing you to reference properties inside nested columns. Otherwise, you can only reference flat columns.

![Form to choose the funnel name, warehouse schema and table](/assets/images/table-and-schema-ff2f4be72fa5152009bf67fd3308a5fb.png)

### Configure filter conditions

To minimize query costs, we **highly** recommend including a filter on the partition key of your table. For the atomic events table, this is usually `collector_tstamp` or `derived_tstamp` for BigQuery, but may more recently be `load_tstamp`.

You can add filter conditions to select a subset of events to be used by the funnel using the partition or other columns of the table.

![Filter rules to select a subset of events to consider in the funnel.](/assets/images/filter-d29701e0f167471467c96813623df43c.png)

### Define funnel steps

Next, you are asked to define rules for the funnel steps. Each step requires a name and one or more rules, which can be combined together using conditional logic. You can define an unlimited number of steps.

![An example funnel step configuration.](/assets/images/step-f6915953b310a86243e2e0463693f598.png)

> **Note:** The **Update Steps** buttonWhenever you make a change to the funnel steps or the filter rules, it is necessary to click the **Update Steps** button.

### Check additional settings

There are a range of other options you can use to configure the funnel. Use the **Settings** tab to update the following configuration if you prefer (or keep the defaults):

- **Order events by**: this must be a timestamp column. If you have added a date range pre-filter, make sure that it is the same field as this one.
- **Group funnels by**: choose a field to group the funnels by - we recommend user or session identifiers.
- **Additional columns to group by**: use this to visualize additional dimensions in your funnel analysis e.g. experiment groups.
- **Max days since funnel start**: maximum days since the funnel's start for an event to still be considered part of that funnel. Set to 0 for unlimited.
- **Intra step time (hours)**: maximum hours since an event for the next to be included in the same funnel. Set to 0 for unlimited.

![Additional settings configuration.](/assets/images/settings-523e3b54a60ec3c88a280df2e062d057.png)

### Build the funnel charts

Click the **Build Funnel Charts** button to generate interactive funnel charts and tables. You can also download the generated SQL to edit or rerun in your own environment.

| User counts by funnel step                                                                                                                                                                                        | Funnel conversion rates                                                                                                                                                               |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ![Bar chart showing the number of users who reached each step in the funnel, with step names on the x-axis and user counts on the y-axis](/assets/images/output-user-counts-ac7f31ea5f1fcebaae831992500f0d25.png) | ![Bar chart showing the percentage of users who converted between each consecutive pair of funnel steps](/assets/images/output-conversion-rates-3588ab1c1a6291f9a006ee8ba287f5b7.png) |

| Abandonment at each funnel step                                                                                                                                                             | Summary table                                                                                                                                                                            |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ![Bar chart showing the number of users who dropped off at each funnel step, with abandonment counts on the y-axis](/assets/images/output-abandonment-b53bfa0feabbaeb163846dbc6128a778.png) | ![Table summarizing each funnel step with columns for step name, user count, conversion rate, and abandonment count](/assets/images/output-summary-eaf18e7f7f49138a6a05440b11014a06.png) |

### Save your funnel configuration

Once you are happy with the settings of your funnel you can save it for anyone to open and use using the **Save Funnel** button. The funnel configuration will be saved under the name used in the **Funnel Name** input at the top of the page.

### (Optional) Output to a table

You can also save the results of a given run to a table in your warehouse in the **Settings** tab. Use the **Output results to table** checkbox under the **Settings** tab and choose the configuration for the output table.

### Export visualizations to a BI tool

If you would like to visualize these funnels in a different tool, the **Export** page contains instructions on how to run the generated SQL and recreate the analysis in the following tools:

- Looker
- PowerBI
- Tableau
- Preset
- Streamlit

## Editing or deleting a funnel

Once your funnel is saved you will find it on the **Welcome** page. You are free to make any modifications to them as you need if you click on them. Deletion is also simple, just click on the ❌ symbol.

> **Note:** If your underlying data sources (views/tables) change (e.g. fields removed) the pre-saved funnels might break, and you may need to edit or recreate them to avoid SQL compilation issues.

---

# Data model visualizations
> Overview of Snowplow visualization templates including ecommerce, attribution, media, and marketing analytics.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/

Along with data models, Snowplow provides dashboards and visualizations to help you extract value from your data quickly. These self-service analytics tools give you templated use-cases for data collection, modeling, and activation.

They are deployed in your cloud and aim to reduce the technical barrier, making data analysis more accessible beyond just SQL users.

![Pipeline showing data flowing from tracked events into visualizations](/assets/images/data-apps-pipeline-dcd6e0fa792a1663ec1ed612f217c477.png)

## Available visualizations

Purchase a Data Model Pack to get access to visualizations.

### Digital Analytics Data Model Pack

This includes the following dashboard visualizations, as well as the required dbt data models:

- **[User and Marketing Analytics](/docs/modeling-your-data/visualization/marketing-dashboards/)**: understand your customer engagement with digital channels.
- **[Marketing Attribution](/docs/modeling-your-data/visualization/attribution-modeling/)**: understand the impact of different marketing channels on conversions and traffic levels.
- **[Funnel Analytics](/docs/modeling-your-data/visualization/funnel-builder/)**: understand the sequential steps users take toward a specific goal, identifying drop-off points and optimizing the user journey for higher conversions.
- **[Video and Media Analytics](/docs/modeling-your-data/visualization/video-media/)**: understand engagement with video, audio, and streaming content, including clicks through to conversions and advertisements.

### Ecommerce Data Model Pack

This includes the following dashboard visualizations, as well as the required dbt data models:

- **All the visualizations** from the Digital Analytics Data Model Pack.
- **[Ecommerce Analytics](/docs/modeling-your-data/visualization/ecommerce-analytics/)**: understand your customer engagement with digital channels.

## Access

You can find the visualizations in the **Visualizations** section in the left sidebar of your Snowplow Console. A visualization can be in one of three states:

- **By request**: if you have not yet purchased a Data Model Pack that includes the visualization, click on **Learn more** and register your interest. A Snowplow Customer Success Manager will then get in contact with you to discuss getting access.
- **Available**: the visualization is ready to be set up. See the installation instructions below.
- **Live**: the visualization is ready to use.

Once the visualization is installed, clicking on the tile will launch it in a separate browser tab. By default, anyone in your Console organization will be able to access the visualizations.

If you wish to invite others to use the visualizations but without giving them access to the rest of the Console, you can [create a new user](/docs/account-management/managing-users/) with the `visualizations user` role. That user will then only see the **Visualizations** tab within the Console. The permissions can be managed in the [usual way](/docs/account-management/managing-permissions/).

## Installation

After purchase, click the tile in the Console **Visualizations** section to begin installing it and follow the steps.

An example tile:

![Visualization tile for the User \&amp; Marketing Analytics app by Snowplow showing \&quot;AVAILABLE\&quot; status and a \&quot;Start install\&quot; button](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAhcAAAD+CAMAAAC+09cZAAAC91BMVEX////y9Pfw6/m55/3w8vXm6OsQGCj9/f5nN7dnObf+/f/6+vrQ1d7z8/P39/hBJHf4+PklLTsTGyueoqhDSldhZnIaIjHT1Nf19fYVHSx/hI27vcEXHy8fJjba3N/q6+0/RVKipavt7vApMD/7+/zCxMhbYGwjKjnd3uGSlp0tNELh4uSZh7jP0dXv8PHZ2ty2uL6NkJhBR1Tz9PW8v8N8gIlOVGDVyO3Hyc2Hi5M2PUsxOEZpPLjm5+hKUV3LzdBRV2KKjpWCh480Okjf4OKPk5qbnqR6fogcJDOdoKZGTVkBL0QXHi78/P3o6uy+wMWXmqFWXWiym9zEuthUWmZma3ZTWGQrMkH9/P6kp61kanTExspobXiytbqtsLU6QU7x8fKoq7Bqb3leY27y8vPMztKws7iUmJ94fYbS1Nbl5ujj5OarrbOQlJuFiZHq5PbW19qmqa9wdH64ur89Q1Hp6euZnKPs7e7Ky89yd4EfTGHg1/Do6OrT1ttLeY52e4RITlsvNkS8vsK1t7yWmqA0YnfRyOLX2NxrdIhLL3+ih9RYXmmo1uyegdJ2TL9ncIX29Pvs5/fk5eZqPblvnbOFjJ7a292NlKRsVph1eoNJT1wHNky04/nj3PHb0+ydjLtsQLpYPYjaz+/WzuaSwNZmTpJnbHf59/2jk8BpcoZscXsMO1Dy7vmv3PKs2vCfzeO9stSTcs2wocmqm8Z6UcFyR72JdqyJjZRCJnjn4POMu9G4vMd/WcSSgLNgSI7g4+fGtOXS196/wsZwRLxFKXsqWG2mjNaNasqFYcd9qsCqr7ufpbJ8hJZ4gJN0fJB+gotud4tQNYI9aoAEMkcnLz35+fnZ3OFaiJ7RwurOxeCvtL+lqretr7RjkKWAa6V4Y59zXZxwWprx+v7P7v36+vzt7/K9qeGTmanBt9Zpl61TgZZEcoc5Z3yFs8l3pLoYRlsSQFbMvOiypctUOobKwd3Ivtuxmdu2qc5kk6lRfpOznd2rkdi6rdKmI8bjAAAjTklEQVR42uzUMQqDMBhA4RSULMU5W8AD5criYbxHkThY/OliN7/vDI+XXnClC351McDuu4thq23k6VrdhnMXw1xLTjxdLnXuYaTjFqpgl/sxji5qSbAr9dRFswu63E5djAm6URfoAl2gC3TBlS6I6IKILojogoguiOiCiC6I6IKILojogoguiOiCiC6I6IIb/t3FtKzvm9ZlSnzYu7uYNqs4juO/ePF76FMl7bNSWm1HKaWwkiJj0myFVQaMMcLLZhTK2AYIOKZjm1CcTTTsLcHUiDiymS3RLF64bNnNEo0xLnEvyWbm68U0i/Fy2b23euM5pwU65hC2Gi3hC01o+Ofc9NPznD5ckDUtykXuVa+Gx0zzXl2BkT0tysXXXmQg79d43AocWEKlE62Yn3miDKL1e3bjsRqbGEB6GycccvU39pzAsmhRLq5qyEDaVcxvFV0WiPIYwj+3qpNG5SnM9AztyScFJSzEg+WS7Zjfb7RD9Ck7sXD7DHrw8GysQnr1XAOggezCsmhRLp5ERnpwmTXkOoheZCX+sd00tntY4ph7zn7IxsnjeDAHeRTz66MLokJ/FRbuDMk38dAqOYT0qjkMYKDfP4hlURa58HMcWj2fSXPBIIAy5xJdLKYy8gjjS3OxnPo3XOiP6GJbpCmwYzVE3Ts6q0fkuzUe6/ut35MP2U7hYl0RB9Nd1AJ4mUkX3bVWt61QAwpjQ4MvVxYqF754LKLBsuGwu3azBa95AmQstlmMxIGRWF9Xszv0CkQNfmt1fMJjG0CytQx3scQMYK/n8MaXOgMjXgBaoa3pdu36lIuh2DaIhmN1+2NOFsUqveZmz14AjR3uI/0HJc2hmD38dDmyr0y70OVD1x/FxdM0Yla6TwHdLgaKaBccPOwkOQpZO52FHh5uTXNhZTeeNdxO6aKCLAqQG4AtLHGRVdKFL4/OXfC9S2fM4EvYS9VOMeKRixdRZBxVRxTRbfI1JAtzU28LjyG5LkURM1BFht1kcdLFMK0+wOJm4btUNQyQo3JK5rwGzUZXpZ0xC7KujLvQ5UPTl+7C3MStsMQ9fdCsrAJGeATwkOH9mwsApDaGkBdpLvawGf38wi5dbDfiwBDtrdhCWoc2Bx3kqGKBRrrLcKqJr2t7N7ElGCyYccHxU10ltAF15PETu12zLoJkKfrZn3RR/Up+nkAGrallH8yV7Ei6KHCyAuij2zEWLOJ4MB/KRZAceXVvEasxSD6LjR2eCWRdGXWhWOjnnrgY1XV9qS58du4sh+wgeQJ4Vr5IHpb4kKrCTdrLgHigeMaFt5Mb6O6VLlStfWSZdLE3eb6op/MYgH5eAJDHl2fPFykXVg04TVcrmhkC8OOsiyFWArvY0qtcNAIFBtdClTvCgHIhV/QDEW6ZPV8oF2toHALaI7WWN8lGM7KyzLqQHH6frLn+px6NCiFLvY7QaN7WBqwiazs6QuQxeJiHVEEn/R7WjzlcPDXjYp0Y5QYkXZz4osdJ+Q7dQiuUC1EjRFZWd3R0BBib7yJPLiMxqB0Kr8y4aLXyePIKoVysBhCWqhCMB0haUy66abStJkvvd5FHD5KZm0m7f3c23s/LuIuPJk2iG7qm6//sYjcZhOgI/QD2hQzSHcQeMqCqgIc7kWoHz2iHwuxZSxtmXZg76S5Puhg3WD9yOumic9ZFfQGAJroDstA8F2rxY2QDrPwx3cUzZNhmszUxolw0AIiJ4XWHyeah7SkXCstwFTtwv4t+NiOVY7iHZMyLrCvT15EDn5tUibuLcdGgzomYYGqT7u0rYQijZAFU6S5scnSsnuTRORfY13kcysV68hjQd7+LOhdtDqCS8TmJhvlvXDSrV7eOLEsZnOm1dBdr6W4ALsy52MP6ElZACdk/46KOdgtQtnZNK0QnNpGbkHVleL/ANzkmVU5CX8wyJby9tfy5EHkNpfH9FsDPHow5uQ3o9vf3prsYZ48X2tPkbUeaC5ly8RvtuWitvd/F0QqD77biNAMD8I37C4F8slB70EUV2TXwnD3lYp2d/atEjU6uTXcxwgjQVjTnwusiAxpEflaOpVy0k69DO0y775n4KsBcwnFkXZl1gfdyckw1ykVOQlvEMqMuqkY0bHQzXPeSwQ1AHZ07LrQwgnQXg3a6Q0VkC0OWv3FR4GTPaY8xzwW6yBG0iaWHPDTaAXOAZOQBF2+5KXKlXLxOrobMz3CaCzlaO9JpzLnAp+RmyAopKlYuLGfI8G0KUqNifn+ILEbWlVEX0RcECEFCuhAwootYpjvSRKNHbbT5tQZZtFkDfMNusqnqKSDC05gpv9klRkfzrRxJHQJcFqhusxBYX03W9xlGGTawByKLi+uBNQZ34WDEIG3XIGrYbmVEjERmFr9GYwxoyKvuOd2VOl/sYAiqowbLitXvcUYON5aQtcOsBkLSL9DHlkNQrREmiy0uaSB3pxjr3ApgV5g0wqPIvjLq4o70kLqOiK+ET1/EMlpZLlKZG3qRauNGPJivwYIFahvDQ7OU5mKBnu8bBLCV9lYsXFkv0tvOnfi7TqS0wFGahTe1Muzi47M5IlPNvfdSOm5o85b533aB7lI4mhnBkioln8eyLIMuDlyW24Rp+gaQkCoEjoQOPStcvGql0emkO4glVUUPlmeZc3H3G8Wi5uaUHtUTpmlTjui9c7qeDS7gHWq2VsbzsbTaGw9iefY4LvT0n32XTKqzF6HrejSRPGnUJKK6ng0uVsqECz06de/GleicDV37w6Q+hkx+KVRAwTApGom7WbFfrJQRF1M3xWt++fepGRe4krxrcfbOzInibqJGXVZqbkRXXGRfj+RCb02ozcE0eeccVAeuJ0+a96DrakKUEFTO5piuT624yL4exYWuX5kWKGQ1n93TADwxmbr57RMeoL7ljmGSTT+h6wu4MGNRDVT8jJUyVOZd6Hryj2M5suQJ4vKVqYs3c0zy+c27+mxy9oZ0MSk+nzzMhe+rk+e/r9AX80Hy/XVYKVNl3gWg6xc/MyUdiKSPzy6Jp6LLU1AioroelYMCxvXpyd910UNc3PphwpH//a8rLv53Lf06out/5igPcrNIPVTTH+my6B+XLz0BXSHSpj44hwVcnLwF4KuTKB90rO92AHAEv/oFaAsC5m4fMNhWPlhe3H4o5cLbfa0N+EVMHBITlu7svMWcFS3VhTpcqL3i8ieJafUhJIWk5k6SxSV57fgYmNGwkIuK7woge/udk9+e/CkX5d9/t+r8LXx93oK3359A7vlX5W9++KlAuZg4/+GH7wSlI9x6R0z8sPJ/dNL6j13gwF/snU1oE0EUgB8uspupiOBBb0LJQZCK4KEecuwt6MHTgle9ePGoZGlvgSJJG0lDD0klhEIIUUsgOWgKbQ21P/khtNiWBi9WWvpfL8XqxTez262BaDZtdV19HyQ7ebO3+XjvZTrb6D2mJwFMC+ZMMcKrstDirghkNcYBnZ96cXMwtdYtoxepMZArHbC2JkNXfxerdEOtXIPuMs48hrZInnvhruYB8pULd1KXoVTugNkiECZ2e5FELTjjYrFfZOJGRQmCqmJoSdGnC0MyU1nTbVO2UKyWHmO+wPHgsly9hddyHoqzcqW7zGZr+szyIPeiqx/rSXv/ApQ6zpbeFiHyjzza9VfSqhfqO71sZNwyF4Opr58qPJJJivSgFVzGwZycxlhzL5ALg4OHXtzv7+LXWeiIXI1A6Vrkkj5zq8y96KjKAO7eS1Cr5WevV2/20s/01WGnF+q43meOnjHrBBt6mstlXgBDQqPmAQxPVpOhiReXy934fq+XGV4ArxRtvU/gfG/xBizXsIkQM8Ui92KMW7PQ3w7dpcgCtqyDQNRhoxfiWK+ixBPmY2UMcSf1a3KKTxpqeHJDzbyAtcjC7YcRM1/Acmnh8mzlPo77H0JXag2w/bwld1cecC9YZO3KHUwu0FatyJBP5YE4wlYvbk/raz6uGs2kuBgw2BezSg4LCR8UNAz+0ou2WiVVql0/9AI/9/ZGxnCcLwGw8lvekRarqYtt3Au4Ppiqrl3mOhUBrqA5RB32efEiLlY+8wlt4PlBeGH6Ma5nir4hrU8RZmQTONukv7gJddw+D3WgMe8vHPUiDn16y3kcx4tR/dmQpWwh2KnCoRZMM77AjjNI9CkIV0Q94Xkt9IKwiJ11hDcXnaJ4JOIKMqqphhehrEskiX2GaGF9ozw8pJ7Mi4UIEFaxr+98EcxNJ0QNUUcVQXwqIXMtkhkXR5kKidKh9eli5BLOOMdHnHBfy80Q46CeANuNJRnYXRwgoyFZeMGe9/E5jyebIC8cSKteGN9A1ERYOMFf/KyvNu7y8EHuOag4z81I5EQh8WST5IXzONa5HIBQVs8V+guFCIveQvksrNHFGAorPBZ/Rl44j+Od44NvisCwQwerxr5bnxaw51lxwvMReeE8juUFdMaFD1PJzqm4IYZoNNyGNgZBBSEvnMixvEjqeaLAV1ybFuWCM5o0jKj34vXPvfi4Ip02Kx+BsMmLd4pglXHc2rTQIhzW8FNLXnyUfgckhk1edHrE6b0p1EDFl8yWMh7FFf/MOIBYrSMr0u9gBYg/7oUMENL/eFYYAo5QQ00Egwmo84JZ8EL6PQBhw/lO9Z3eZo6bEaTBE6vkhZNpvY5ocbFVMZ2EpgRd5IVDadmLEO5voxe5M4yRF/8urXrB9vV9zq8MTjVfxHY38N23yMcj0W09gLyJvpQOtiSDgXSAXyaj0bRvXZIC0WERPohyNskL27xgSx6xsZlRwYQZQB2sNS9moumYJC3u8vEXHIoA98L7UurxSwa+6Lq4LI6MrHt3pIBX96JncgQJkBe2eRGadiEeT/CuFfBmq15sLe7OYKYQCz15YAbqvRiOrvuEFx/4XT1HXvipjtjqBXvn+hlKIyx7EZgfWfTzxd4QcpiBei8ONl9Ghw0vAtEB8qIhdpzjK/DFFijWsOjFl0lpZD6ADcQeFhOfGaj3IpbeljZ7uBc+/+Q8Dkwv9uaQGHlhjxcMnoUVpFmiqE8YnmcWvOBd5CRmgVfzbyTfFzNQ78XMXkzaTsdwcnNmp2dv+8gL3wBC+cKufMFCfeZ6W8sb4v9fNPdi2Oufm/PxPDHXMzwfMAP1XvgxLWx5d4z+YtNPdeQIe71gqwWXwOM61ML1KyuQ+Cpr7kXP5ACCuQLzwYb/x8CRF2jBB4xt+Q0vNnzkxRE2e8GePe9shUQCq0hTL2LpGX7Z2sTh3vzMDwHDi1dI7INPTy0j+D31VWAnvYFebPMZ44aX5IV9XjBg1gHAlwUvdtIxfQ8Dl3YDxz8EhBdezvouZgl978vn9c77B2LoBSdt3OAnL+zxwoBBi9A+uPM4hhdAXvz7kBcEeUE4wgs6x/fX8ie8oHO/zuNPeEHPCTgPS15MyHAKyBNAOAVLXoy1wynQPgaEU7DkxfmJdhlOiNw+4cTfKf9fseQFnB+bOHdCJsZICwdR7wVBkBcEeUGQFwR5QQjIC6IR5AXRCPKCaAR5QTSCvCAaQV4QjSAviEaQF0QjyAviO3tnstNGGkXhs7q1sEB2YTseA9h4wmIwYGRsGYPNFMQoMQgS5ikEghiakEhICSQSC3ZZROo8AIu8QPb9Ar3pZR6m/VeVr1100U2nG+ihjhRUw63/DvURV+Q6ipFMLkwZyeTClJFMLkwZyeTClJFMLkwZyeTClJH+i1yMmf///j9Pt+OiZW7uoEmCkR6TF4pqqBp/rKVV6DU+TD/jFrplhreN+D3ZPFB1FALsDoOIBYrjL4oz3KUCEy6D7YHovXLxis4yI/7kXXBh77xYfgAu7OHbcNHUZcV3iDPcoTrkJv32VgewPvLhnrloRsP7yODfz8UOLQAPwIWl4TZc1FMVvkOc4S7lurYdTPLBe+UCaKUUgNeJ3n0UJaXyi1UKF17PolXctdqBwTllGNsv8wIWjn0UO07MASFPIqpw8SkxkIWi1HN6E7NoV3AgMC5+topD9T8BVZvJGpcug8i/4HlyCuDJxqOvE7UKognP0xIXAxupva/b2O/vV27z8mby1anCRSr/VZS026px8aLm501oan6V3N0UXCzPJTfbgQEHJby8xzr5NjixDsASyie2RdFWUec516l0ITIEYqfqbCCtDL5afrUPoaYYgGzslEfIRTx53To4pmsO1vPi0NQFtpq3LlAb+whgawPxWABwLeRDDRDb2zF/d2x5NLbN4+fsd84Fgglgb2Rmyv0cWE7bHJkRb5GLtsYp36yEGirYJ909ErAlT6755iWOLZ7K2AbxgeyTv/hXgXnnpV3eg9BuD3U7qvkKNVBM2e0CgmdAlXyAQMEf9qWbyxmErD1+R5u8ANjabJc29ycgNXS2FgxqXNgKZ5d+Z9fQTCO9B2ozZ1fpSAqwZWzhgm9cMKH+yfkyPTQPRbXFPOkgxRE9a7xqdGbxpY3CA7xXUtzmDPuDWUhhv6NbHsBjOgLG5BWuU+lCrP6Y3qmzwaU88zkt10NogiTgNeV4hFyELSMvPdY1Z+mmmbORc0iT7vBUhg7wiEIA3P3ooBY0DaUd/uBHse11+AqOWis94fFz9rvnovsSzY0DwIRzGXP0GNLScDF7WEKKNlBDNcAmLeAp1QPjtMKxxVMxYMx5COwPPUOfCPxRrobQNj0FX6EGCp2SF01OeR9eqsVU2oXjka5yBqGOYBMQ/iyGGYDV+RKoW6pG1VKJC7sVzW7aB9ocQGfaVRxxwVLEpRmWuiXmot22KuE9hSCUdMeBVYqja8mFBtue9jnCe5rsbQFURb6gV34M5N1R2J8BNU5XRZ0xKKvzbFrpAJgjPRc8Qi7CdlYLfXMeZzFkJoMF2gUWrnPxbLYBWXdebKufI4ILHiZnv3MuLu3ip+XjOHXAS/3taLCqzxfSSL/66W/1HcLjrgIQnOFY9dQ3GlefL9bdjiwkq8Rc8BVKoKq6JPbWunvxsx1W9wCUKXEGTe3xr2RVHxl6ZnBK+fLzhXq0exiAJwKXLPbqKace7qLTEhchQU5DRL3pheHy80UgPmVXueA9TcvUD+B8F3ZxKFrcG/Bb0DOvq1PlgmdzKFeJC/Vc8Ai5CFGdvrk6McWLWMOh3A4EaEXPhWfkfBQWi44LHiZnv3su6r4AoTYaitA5kPD7wgtQs8PZq03j8wx6xHixmuFY9VRCdqlcwNtI7yasYC74igou8j9gaeVVHdq6sEGKGssZFA3MUsRf4mKyE6/p/DoXMz2CAj9SCpJxWlEPhyhX4uIlKXoGoRGPxoX17ZBs82lc8J6mFOUgNDoiOETjF5y4vc1uL9fJXJRnM1UHAy60EXIRzAU3N1oEQFGPHQZctE+7I4dN0HHBw+Tsd85Fs7MeO875PrRQBwDL+bS8d52L4TWEC1B+VzlWPTVAWf73SFMy8q7MBV/BXChzkwMv5Au5qRiy9aioi3IGoU3aqsWTCi4uaPcmLh7RNwWHcfXwOF2UuNiSQ2LtOIQihxoXP/p+csFR4oL3ShW/hyK/CK8WdFyuHtgauE4DLubPDLjgEXIRzAU3538ORWsZIy6A5YNOt1fHBQ/z/rh4MxTHIl0Ar8W97gIwHazkIgusO/fwlfoAl3uQY9VBhegAkDKreHq4LiYeZS74igouRiP2TsA+ZZOKJ+bFCKVyBqE3EQlIVHBR7RPIfTHiotp5BSBPzbAdiiNuV4mLcaoBYIWi7jMAKxRHWqDXo3CxDt7T5HL/CKA/gc5ZhbFWYNE/9RZcpwEXvaLBrMbFEUXFdTkeIRfBXHBzw2kArVejexQHxsQzm1wPWGWVi+QuIM1Oa1wMKlzwMO+Hi47U3IwzBTTR/PbmLO2inrayr886K7mwpz6tuXfQHJmJ7jyLRDlWG1RdIdXyhp7hsbz6qWXS6WIu+IoKLjBPr4B+WgWQH/H05YYrMght0cv9iRHKMhcYdH7rG5CNuECvc/fEG3kO2OQP0c2hN/zcKb0Lvsq2Bp9CqJW6+o5sFMfl0ELTj5QBcrRby3vSlBdCSX8iXiPXI0X500d1w1bA6qeNcp0GXIz5Li9y3RoXVuezTx0ZyvEIuQjmgps7ojefvMU7nx0Jx1sm6QDobNvITZLKxbS/dWxT7lW5mBmOWgUXPEzOvjJ/d1yQXPdc9I6aJYoMyHngZYQiV/EKLpwTToocAWixkzycK8dqg9opHp4Xz+7jGfJNhsBc8BWVXOzSDhClTQANXU4amrZyBkWWQRulu2i8zIV1Wib7qiEXUu8Q/TIoAba3wySvtTMXOHG4ydZfuh1OCorfzB2HkyavZGsRBVrjvWpaUTMnfWSbEBzNkjMcQFFfCijXacAFQr+Qz6Nxgfo0+ZOU4xFyEcwFN4eDCDkPrcC4jUbeCi5SneR+7le5cD1zUiGpPXe22mhfcMHD5Oyes3v53mxMgqpjCXpJWe1IVVVlLOukdNebrWDxFcbihXmDD9bimlyBm1c4RSn3tUyWU7AaStuuUlCtlfcWKQpVo9nSWYtBncbKNnTQOHhyvKEvwqC54wZtAUtABfPEUhE2BpZl7IZhpuf/ld+n/lu0eoXv1djnbUR7Isv4SxJcfIfizn2TizuUqxrfKykvRyj4Ex6EC5z8O9+/+F+oKpSrxl9Uw9Mx/IH+S+/lmPoHyuTClMmFKZMLUyYXpkwuTBnI5MKUkUwuTBnpf8/Fd7gybB7cQs10gL+qAeVLzDvwiNxC/xkubvZmBLoew0ALNTe6Mjji/rjgFrha4d4w4mKr4ztWZV/I7fUf4eJmb0aUWmGg6bobXRkccV9ccAu6al0w4iKY/O7BuHB7PQgXDSFPLK6zJAgzyHsvOx1U88VxrLkydjH5IYvxfP1yhUfiaPP4Q9cn9mYIsW9EDWn10GqNOJjoypVTnsTqgrF9xfeBj/XJA6vOOiKkRrCLA1IukX+qcFGO0rlaPljmPE8lEbk7OBHXuHi/CCD3ASVzBhfC7g69vUSpVm2g6N4QXBS7rgUbPVRjB7fPqxa1PdFe9MNYrd9+PrJwHm0wR4vbnpTwiHAb7Lq5QQ/BhbTmvrT5NyosCVvCDBKcLzsdajNF80WaohzbQum6HnfhR/+k3zdW9kg4zs560lSveTNUaY4LLSQ2RcNvgFf0bkl+wil3HLN+Rwh2B5Dy2edtP9SWrSOq1Ah2ccBDdXbqAmxLtslyVKWr5V1dt3wFIOye+tyocRHuBpDwoWTO4ELY3aG3lyjVqg1YSZRbKKxFGrP8IqZq7OD2eVUBmrzUWVyiO9Pj/gGcRx2Mw+bM7HZQC7fBrpsb9QBcvJSjsLR1ly0JY75VYQaZLzsdBuU+YJWiHNtCzwEvzQofx5OyR8LhywGdU/rPEeG44BD1b+bX8hzwfKiZUyqfEoKLZf8VcBz5wtYR3ecIuzi8tAIkaOdaVKWr5QPQO2RBK+0CBzSn50IzZ3Ah7O7Q20tEtVoDKhcOIOt7y1wonyPcG6+qcEHvgZ/FBUfUwnnUVR0jKaDIBbfBrpsb9QBcfF4DMCFb2JLwTWzgh+my06FgB4qdRzlWeWncIosR2NfKHgmHXZu7notqcIg66T2bBWK4nJK56KBzwYBPYutIJRfs4kg6LcB6rO96lN7VIt7LPnS7+PmCuSiZM7gQsQ6L7SXMRbXGhTKY7oKeC+6NV1W5sIoz4+LKAc6jcWGHwgW3wa6bm/QAXEhOUhTnVwZfyusqF2zj8CVVLjhW4QK+hBhTmD0SasMx2YALDlEmPUWK5jglc5EQtxEH9IJf7azkgl0cnRoJ16P0rpYmSqHHDgMuNHMGF8JcsKHkZi72fHouuDdelbnIUQqQaILz6LngNth1c6PunwsMzT8Ssla8/L2jcsFOh19WVS44VscFeyRu5oJD1ElfFpRlAr/lYoJOAXTJ1YZcsIsj/IMxF+xqYS4cBSMuNHMGF8JcsKHkZi7yET0X3FtpVQMuOI+eC26DXTfGehAuhoMSACv4JqVoDpAK02WnQ2cjgF2KcqyOC/ZIVHKxruOCQ9RJ5ynKKXVcqAZMDNfBkAt2ceSpFuhzPL4exa4W5qJLQP5C4yIZBJD3oWTO4EKYC5295Ldc7IqG37HRQxg7uDde1YALziNWZS64DXbd3Kz752KcHBsth2WzSMkMMl92OrTSnjBfRDlWxwV7JJgL1Zvh6WcuyiGWobcfUetvW3gxEQyUU/YP7S8LLtB5lqqdcC+UuXg09QKAGsEujmNnz9OmzsbR61HsamEujn3hltxnmtNshlvRA98ISuYMLoS50NtLlGoruYi87+ulRTZ6qMYO7k1b1YALziNWZS64DXbd7M7BSA/CBTaD5H4Xr+Ai3kbyfGa6wukQc5LtkKIcq+OCPRLMherNsH9hLsoh6PJFgEdtRIXxipTRNA0oXDQX49JzKHPhpY8A1Ah2cWAjTXL3C31UpQOGuUAoSM6Xsjrv0aSNCs9GUDJncCHMhd5eolRbyUVslnxfwUYP1djBvfGq17ngPMqqzAW3wa6bnmEY6f65YCOFXieu5ZF8pdOhIQuPbNXHGhs1eMlTud4wZH1Z/KwKQK/saClObx15E9RFsIujud04akyC8cq8uN6coS9Eby/hass6tmgTsrCxg3vjVQ3EeXh+3EaF60aSYKSH4kIvKbwoBZKUAis0OYbXwR78KR2k2/E36Kz+llGm7pgLbA4NuX1blVBPkZ/sO/hzCuDv0Mkto0zdORewNIUC0Om0Y1+Cqf+G/m3vX5gylMmFKSOZXJgyksmFKSOZXJgyksmFKSOZXJgyksmFKSOZXJgyksmFqV/Zu38WxYE4jOPycCDccv6uvBdgOd2+Dqt5Ib4A+0DKdGl2gwQhxRZiIMTCVL6BFBY2/umsdZdtrrjT5ATDbGRT3GTh+RQTRcsvmXESjAm7IBN2QSbsgkzYBZmwCzJhF2TCLsjEVhddm3gX2H1Wuuh2B/0Ha/oDlnGXjS66g94LLHrpDRjGHTa6GPRgWW/QoVoWuuj2X2DZS58njHo2uniAdQ/sop6NLn7Cup/soh67IBMbXVhfdgI9dlGPXZAJuyATdkEm7IJM2AWZtKgLrfEhdvHFNe9CiSh8hF18cc260Eop8ZUorWDELr64Zl3IWXnQuDVELW+EOnGEUDns4rNa0YUSpZQ+v9BK1E0UkZ/myRBYP+PWaF0Es5BnVIQOriZjTGXGLj6rFV1o0f/WnZUuEtnMknQLLLe4FYiDMyfyULF32UXbND1fQPsioqtdjHMAW/c5TrMYGM2iEJjHzvF42Ipa46/XeP4Uh4fg/MY7RA4wy5bxCMP3QLOL1mjShSrWFr4SH1puwthJ7AEIM3EzrHN3LAuEkss+yWUcl/OII2N3KRFGJ3eS+t7ETbP1PFtm6YRdtEWDLrSocvBFF8PV81hOwQjFPBLtR9i481Cy9VM5j5Rd7Lx57iOWA6JsfZlH9P6AWDS7aIlGXfjloESXw5V3nMhyWnSB4SHwxQnPCd12EQA7F69pFo+u64t1tJGIXbREw3lEwxflF/OJwq3X3PUuXbwvs6SuC+jtMk28SxdhfkrYRXs0X3eKiF9dd47SHYCNhJcuJrmHoKaLt9+YJ3K4dLEQB2/sojUadwFoff3JerVLF7+j5clDNp5ikx6PubwVXbyLCitdBBI5ibwhSd+fZrKY+hKwi5Zo2oW+lKFUdV8ryeWUOEDsyjDcp+5G4qKL+UQWlS68xJUsAl7HMvU2bprIhl20RMN9rTOgOFQ8lYXMzzFUPzB9uTwMh7w+0h6Nuig3wP1iO9yM182+NF5nJ96XQ7yPj9gFsQsyYRdkwi7IhF2QCbsgE3ZBJja6WMG6Fbuoxy7IxEIXnf4jLHvsd6iWjS66q2+w6htPF/dY6WKw+vX4w5rHXyv+3+89NrrofLf9/+DfO/R/mLto5wMFOvRZfP4InbELMmEX9KddO7ZBGAaDMAoFygrufikDef8FsgcxEZaLa6HJezN8uuoSXZDogkQXJLog0QWJLkh0QaILEl2Q6IJEFyS6INEFiS5IftpF94zisvWli2oPGFotXRxlMBi2OmYXZxh7NWWwtdrPLGYXYzH6i7vrYy1mF58wYHh+u4CFLtAFukAXXHTBX7wBTc5qwpvS0SEAAAAASUVORK5CYII=)

During installation, you will need to provide a connection to the warehouse you would like the visualization to use. It will also highlight any required pipeline and data model dependencies.

The installation workflow will look something like this:

![Install wizard for User \&amp; Marketing Analytics showing three steps: reviewing enrichments (Campaign attribution, IP lookups, and YAUAA enabled; Referer parser disabled), enabling the Unified Digital App data model on BigQuery, and selecting a warehouse connection](/assets/images/sample-app-install-8c42522fc02777615519062089af612d.png)

### Data model dependencies

Generally, visualizations will depend on data models. If there are dependencies, the installation flow will highlight which models are required and what models you currently have [running via Console](/docs/modeling-your-data/running-data-models-via-console/). It will also highlight any properties that you need to enable or configure for these data models.

> **Note:** If you are running the necessary data models yourself outside of Console, then you will need to manually check that your setup satisfies the requirements for each visualization. These requirements are listed within the documentation pages for each visualization.

## Warehouse connection

To install a visualization, you will need to create a connection to where your data model output or atomic table resides. You can do this as part of the installation process. The visualization will need secure credentials to fetch the required data.

You might have already set up a warehouse connection for loading and/or modeling the data. Visualizations, however, require a different connection, as you will often want them to run under a different user/role and with different permissions.

The connection process will look something like this:

![\&quot;Add a new applications connection\&quot; dialog listing available warehouse destinations: BigQuery (Prod1) and Redshift (Prod1) as active pipeline destinations, and Databricks and Snowflake as other available options](/assets/images/add-connection-d2b0cce3fe5e1fd0f0f0e921c721adf0.png)

Once you have selected a destination, provided the credentials, and run the suggested SQL script, the Console will test the connection. Upon a successful test, the visualization will be available to use.

## Using the visualization

### Is it running?

When the visualization is doing some calculations, querying the database, or otherwise still loading, you'll see the following gif in the top right:

![Gif of stick people running, swimming, etc.](data:image/gif;base64,R0lGODlhgACAAPAAAAAAAAAAACH5BAkUAAEAIf8LTkVUU0NBUEUyLjADAQAAACwAAAAAgACAAAAC/oyPqcvtD6OctNqLs968+w+G4kiW5omm6sq27gvH8kzX9o3nKcD3vA5s+Ia9oNFATAKOQKWSiXM6oTbpkzqzTrEx7ZX78ibBYfGYzDKf0So1kZ12/+Ar+Zy+s+Pr7j1f7ffnFRinRegiddg1pNho5OMIKbhGZpXXl6N3YrdlwvlF8ik6SioZWoqaauap2ur6xvoqm4oya1t6easL2Lbra1j2+5sp3ApWjKmI7Bghy1zh/DwRLd38Wi1Bje2gvc3Q7a0AHo4wTo50fb5gfs5O7h4O7y2/TY9tX40vrf/Mz+wfKZ26BAAbFVQmcOCBg4cYEnIYCKIfiXso4rFIByMcnY1sOKLxWCmhwgCWhCQaufDkN5UoS67s1JKlOJkKXa6jOdDmTJgjdRLEqc5nOaDtiJI0Gs+o0KBKkc5ryrMmVFA9p1KqGnVoVqZb0XV9ZxUWyqNflxYt67ReWEZjyVL9+TUp2rhP577NudZUTLtXpfIVu/duSrpq/7Id6zRtYcAvGQfWy60vYsgPDre9jDmz5s2cO3v+DDq06NEnCgAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJpurKtu4Lx/JM1/aN5/rO9/4P1AGGxGGwV0wSjzmlE8C0PZ/R2XRahV2vWdcW22V9qeHV2FlWnclp0xrddr+T8dN8WZff8/o5v3/2BzgmOMhVaAiHaAe2qEbnGCk56XBHSLlgaYl5oOmJ6Rk6GUoaSVrqeIpaqLoq2OqaBxtbNyv6eljZyprbAMvL5vuLG8wwTKxofIxcBLGM2Oz8zKm8S/0wfZ0wq62r2i1sDa6QPc49Ti6O3lkObFR9Opq8rc6sBB/v3kj/rb/PXi/Om0wB2/jh10/WQQTtumxCaGsekocMI2758QmixfZiNxbi28ixxpoIIC91NCmtZC+RXyiobCkFpsuX/2TIrEAz5IubOHNK3Hkxg897NIJuGIrHptGjSIuubPLUS9STNaVWpapTzFWsP1tM5UpUadaYW82U1bA05dhHa4V+JXg2RVoPc721RVGXQ96PkMTetbAX7t9EST8EFtzV69sLh9PFlbu4Z2SNff0OVps43GXChd1OphzNaWNsPEl+xotS8umKj9mOdvwadGcrqUnXlhAbciDbt3Gv1j0QoEfVmbUMRfvbtUqmm82+7NDaOEjDYSdaDBEayPV1wmlxz8dd88jwvnOTP48+vfr17Nu7fw8/vnz4BQAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJpurKtu4Lx/IG1PaMc/a+537FC9Z+xIfwWEwmjkylkgl1FqFN6Y9atc6wWW2Mi/TKwELxmMwzf9E99YvddrvgQ/kcbofh828yf431FxK2ECjowcVAeKgDxsjC9phCJ2lCV1c5eImZ2bEZ16nxmRaaMRpUenFaljqxutja8NoVqzAbVWt7i5trsJuYOxuAVivsS1xqfBDZqYxAKfkquyko7TDKtxpxasctge3m7foZTq56KUdtCq2G3ojcvofomC4PQpVtr0ma594ryu4fBn8CB+orCOQgwnHMYuH7ppAGLxQRn1WkcFGiuv5p8Axu9GRNUcOE2t450+UH4y6AvwBcSwmx5bmWnFAa2kazJoScoGxOfMnTZUxgPn8uuQkUqUV6RpQmpXXUaFFWOB9yhFoV6zKpS7U+hRUVLEOvw8g6HUt1as+ZadXq7IpKo1ircuMW8srVAt1jYte1hcvP7T1aeT3+5Xu4bGKWacn6tRsWMuAbIrr0BXlYquPH/DZzXqs482WTpEZjDrx17uJ5cU2TXmvUM+MesmdTnnw7NeoRGU/Dhur69V4SYGOv/tCxRHHgx1kTtfTXuGTiMKFPl76bd3Xl0ZlP187U+m7soCtvp37de3bz4bmnX/6dfXvw49WXH1wmuO3ciK/fx0fOin6fvaVZc3VRVhtboBX4n29tJEhSfH1BqBJVFGa1Xn9CiWSgXosUZliGEYoYImoXfnXfiCR62J2AO53IYYcYZgfiUDLOuCKO/OnmoltvHZjjVZed9WKNJTYo5HHPFQmjjin6uONdRPJo5ICmzZckTRQF9WOWK00SlIK33CHTkeK0UOZ+H5FZknMEnWGOfOf5EJB7VeKw5EJ67slnn37+CWiggg5KaKGGMlIAACH5BAkUAAEALAAAAACAAIAAAAL+jI+py+0Po5y02ouz3rz7D4biSJbmiabqyrbuC8cyBNT2jHP2vud+xQvefsSG8AgoKhFI5HLZPD6LUefUV5Vec1nhFtfVfmFh75hc5p3R6do61na/X/HknJ6+s8P6fbUPlwUo8zc4I2aYqLjI2Ehk5rhSGGlSRilSd+lRJ6eZwdnjCQQaJCpBimi6gNqkqsAq6GoAG6tKy2d6aymqi6vZu3sJ7BuZN0sr3JWgSzm5zOwYxTAc3bpKzSh9ja2o/To8ZOj9DB7eN34AGtAGiH6cyXnnvp75Hj82f5+u/pVfz8TviruAAPU9GWiwYJwtCBc6+HfQmkJjD9gJtELOYoTpYFMw7oO40VlHSBMpjiLZL9Q3hxtUZgMpa0LCmCFh0qw48+ZDgjp3kuqJExXQaciGZmRltKTQpPSgGf3DsmcsjVLHUb0p0iTWhsqqSlTq8Y1LCrWOfj0zj+hZs6lSElsZFm5chjBFbpt7sZw5I2vz6rVDo+/Iv4CDtqX795TgwXoV4/XLLfBjyL1kLkbrtObhdj/JXhanVTPKX5/5TvZUGENappI3szY8+nXr2LJ9uq59lzbu3KV2i+7tGzbw4KZvE29qnPjq48h1M19+HLry0sGlV6d+3Tnz5mO3y+3kvWL48eTLmz8PoQAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJpurKtu4Lx/JM1/aN5/rO9/4PDAqHQ4DRWDseiQalc8l6PoXSKgBllQKz1RJX2/taRWLwrjz2oM25dbfjdvLibAxdrrvXK3pou49nAeiHM6h0YXiVl4jEZxjG6Dj4EzmRSARo+cgUoCdxydlEFwEa2jn6UGp66gYxuYqA2vAKeyDL0FercLtwp9sb5+D7uxs820oMjHz8lSxs/PzmHL32STjNDI0Nl7vNseltpxquWUluzdh4npp+uJ7drv4eGz81b1u/R56fts6f5e8fwHACm20rKAYbwjLOulFD03BZOYYRDYqzOI3Lt/aB++x98HgvpMiRJEuaPIkypcqKIDVw3NDPBUREMwVRTDEMnUR22khk4ukJKK+P4/AVFQWO28KlP2EyferwItSpPSVRvXqTJtatGjNw/RrTZldltBKYKzZWa1iyQ9EGfdhSbFx4NUm1xSXNZSAK1dRinCjvxlqpc1caPow4seLFjBs7hqTP7169kXFm5VsXc0IVb13dxVs1RFO6oc1GBXGW3lFWZTeCBav0NWzKsl/Trv31Nu6tundf7V2Y9WjVrYkPBhzcaGfQy9kmt1vZbV+d0z1HB0w4rebNVgPbOC73+ePx5MubP48+vfr17Nu7fw8/vvzGBQAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJphPAAuoLI+3cxrZJ5+7Ne/q/6wkvQN3wSCn+kEyHEtiMJp5LqZVqtEaxVS2Sm/UewTnxlzwzn9HqNbk9RgfhPTY9jr27n3omv99UBKiVNmh4iJiouMjY6FhX9hijJJnCVVnyhhkit/khx+LJARoqmkFaY0qEOqcqwVrq+gobK+tEW2u7gJuqu8ub62sA3Cs8TBxsi1wovMys7PysGh05TV29eQ1Vqf2H2N35jXyMKg4sw3o4jl5ueK4Aq07bEO+efts+eI9PKp/PUM/eP3j7BPYDWFDcAU0EBzZiyO4gNzC/JE681DBcNvCKGSFi8kgO5MU8HUmaEmnHGsYpHF2hXOmyZURKul7S/JjM5raRrUJSqdjlUZiZ3ohKY7TTqKCSyRTdVJp0ITakRaXCtNrUXFWsWwMUo3qV69OFkjSyNLnRLNShZR36HPsw4NmfQuchROs07Aq9+vjuxdsX8CrBd2RuMFzYLwbEbRiPcutH8WHIQxx/ogyJ7omEeOBmwmxDswrOmaOOBv3C82mLQkzDkNt6amjYpXvOpm1srcjcXnnxZopa1rrfvd8RL47ruNjg0HATd36ceW5Qyh+krE5PMva3Qbfz6+79O43wSdiShyD7PPqv6tsbKgAAOw==)

You may particularly notice this on visualizations with multiple tabs per page, as the tabs will load in order so the last tab may seem empty until this processing is completed.

### Configuration

Where the visualization has some configuration requirements it will also have a **Settings** page that will validate what is available to the visualization, and provide information for steps to take for any unfulfilled requirements.

### Chart sources

Many of the visualizations support the exporting of the SQL used to generate the charts. In some cases, there may be a specific download button, but otherwise look for the ![download SQL file](data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMTYiIGhlaWdodD0iMTYiIHZpZXdCb3g9IjAgMCAxNiAxNiIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZmlsbC1ydWxlPSJldmVub2RkIiBjbGlwLXJ1bGU9ImV2ZW5vZGQiIGQ9Ik03LjcwNjA3IDExLjI2NkM3Ljc5NDMzIDExLjMwOTEgNy44OTM5MyAxMS4zMzMzIDggMTEuMzMzM0M4LjE0NzI3IDExLjMzMzMgOC4yODM3OCAxMS4yODQ5IDguMzk0NDIgMTEuMjAzM0M4LjQyMjMyIDExLjE4MzYgOC40NDg3OSAxMS4xNjIgOC40NzM0OSAxMS4xMzg2TDExLjgwNjggNy44MDUyNkMxMi4wNjY4IDcuNTM4NiAxMi4wNjY4IDcuMTE4NiAxMS44MDY4IDYuODU4NkMxMS41NDAyIDYuNTkyNiAxMS4xMjAyIDYuNTkyNiAxMC44NjAyIDYuODU5MjZMOC42NjY2NyA5LjA1Nzg3VjIuNjY2NjdDOC42NjY2NyAyLjI5MzMzIDguMzY2NjcgMiA4IDJDNy42MjY2NyAyIDcuMzMzMzMgMi4yOTMzMyA3LjMzMzMzIDIuNjY2NjdWOS4wNTc1NUw1LjE0MDE2IDYuODU5MjZDNC44NzM0OSA2LjU5MzI2IDQuNDUzNDkgNi41OTMyNiA0LjE5MzQ5IDYuODU5OTNDMy45MjY4MiA3LjExOTkzIDMuOTI2ODIgNy41Mzk5MyA0LjE5MzQ5IDcuODA2Nkw3LjUyNjgyIDExLjEzOTlDNy41NzgwNiAxMS4xOTEyIDcuNjM5MDIgMTEuMjMzOSA3LjcwNjA3IDExLjI2NlpNMTQgMTIuNjY2N0MxNCAxMy43NjY3IDEzLjEgMTQuNjY2NyAxMiAxNC42NjY3SDRDMi44OTMzMyAxNC42NjY3IDIgMTMuNzY2NyAyIDEyLjY2NjdWMTEuMzMzM0MyIDEwLjk2IDIuMjkzMzMgMTAuNjY2NyAyLjY2NjY3IDEwLjY2NjdDMy4wMzMzMyAxMC42NjY3IDMuMzMzMzMgMTAuOTYgMy4zMzMzMyAxMS4zMzMzVjEyLjY2NjdDMy4zMzMzMyAxMy4wMzMzIDMuNjI2NjcgMTMuMzMzMyA0IDEzLjMzMzNIMTJDMTIuMzY2NyAxMy4zMzMzIDEyLjY2NjcgMTMuMDMzMyAxMi42NjY3IDEyLjY2NjdWMTEuMzMzM0MxMi42NjY3IDEwLjk2IDEyLjk2IDEwLjY2NjcgMTMuMzMzMyAxMC42NjY3QzEzLjcgMTAuNjY2NyAxNCAxMC45NiAxNCAxMS4zMzMzVjEyLjY2NjdaIiBmaWxsPSIjODA4MTg3Ii8+Cjwvc3ZnPgo=) icon. Click it to download the SQL used to make that chart.

Note that some data is processed further after the query to get it into the format required for plotting, which may include actions such as filtering, pivoting, etc.

### Help

The visualizations provide help text throughout: keep an eye out for the help icon to get more context or help in using some functionality.

### Log out

If you wish to log out of the visualization, you can do this from the sidebar. Note that this also logs you out of the Console.

---

# User and Marketing Analytics visualization
> User and marketing analytics visualization with dashboards for acquisition, traffic sources, retention, and engagement metrics.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/marketing-dashboards/

The User and Marketing Analytics visualization contains all the visuals you need to perform a high level analysis of your web and mobile performance. This includes dashboards covering:

- Reports on user acquisition
- Information relating to your traffic sources
- Insight into user retention
- Measurement of user engagement
- Deep dives into technology and user demographics



## Requirements

- [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enabled

- [Campaign Attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) enabled

- [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) enabled

- [Referrer Parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) enabled

- One of:

  - (**Recommended**) Running the [Snowplow Unified dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) with:

    - `snowplow__enable_yauaa` set to `true`
    - `snowplow__list_event_counts` set to `true`
    - (optional) Conversion event(s) defined, with `snowplow__total_all_conversions` set to `true`

  - Running the [Snowplow Web dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) with:

    - `snowplow__enable_yauaa` set to `true`
    - `snowplow__list_event_counts` set to `true`
    - (optional) Conversion event(s) defined, with `snowplow__total_all_conversions` set to `true`

- Access to the derived tables granted to the role used when setting up the visualization

## Usage

All charts have help text to explain any definitions used, and have the SQL used to produce them available to download by clicking on the icon in their title.

### Filters

The data queried and displayed can be filtered using the filter in the sidebar. Note that these filters will apply to all visuals across all pages, and changing the filters will cause the dashboard to refresh all visuals which may take a few seconds. By default the last 30 days of sessions and page views are used.

The comparison option allows you to compare two different time periods, we provide default suggestions such as the preceding period matching the starting day of the week, but you can select a custom range. Note that it is possible to select a shorter or longer period for comparison than your main filter; this can lead to unexpected results in charts as for example bar charts will use the full range, but line charts will only display up to the main range. You will get a notification if your ranges are of different lengths.

Some charts are only related to new or returning users, not both, so depending on your filters these may be blank.

### Settings

All configurations can be found in the **Settings** page. Note that all these settings are global for all users, meaning if you change them they will be changed for everyone.

Configuration options:

- Select the currency symbol used throughout the visualization
- Toggle between using campaign or channel for reporting aggregation
- Use tables produced by the web package instead of the unified package, and select which tables to use

---

# Video and Media Analytics visualization
> Video and media analytics visualization with dashboards for watch time, audience retention, and in-media ad performance.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/video-media/

While 91% of businesses use video as a marketing tool, few businesses deeply understand how their users interact with this content and what effect this has on business metrics. Collecting vital engagement data plays a significant role in understanding your audience, optimizing your content, and measuring the impact of your video and media marketing efforts.

Our Video and Media Analytics visualization helps you visualize insights on your video, audio and streaming content, based on data generated by our media plugins and [Media Player DBT package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/). It comes with the following dashboards:

1. **Overview**: get a high-level view across all your media and video content performance including "total watch time", "plays over time", and "top-performing content".
2. **Content**: dive deeper into individual media items through an impression to plays funnel, audience retention chart and views over time graph.
3. **In-Media Ads**: improve advertising performance with key metrics like click-through-rate and see what content returned the most ad views and clicks. Know how far viewers reach through your ads with the Ad Audience Retention chart, to understand where in the ad viewers may be skipping to improve ad creative performance.

### Requirements

- Running the [Snowplow Media dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/)
- Access to the derived tables granted to the role used when setting up the visualization
- Media events tracked using the media APIs for the [JavaScript (Media plugins)](/docs/sources/web-trackers/tracking-events/media/) or [mobile trackers](/docs/sources/mobile-trackers/tracking-events/media-tracking/). Note that there are multiple JavaScript tracker media plugins available. Implementation of a v2 plugin is required for this visualization to work as intended.

## Usage

The following steps will guide you through using the visualization.

### Configure

The **Settings** page lets you choose the schema and tables in your data warehouse. These refer to the output of the Media dbt package.

First, choose the warehouse schema where your dbt Media package produced the derived tables (e.g., `dbt_media_derived`). The visualization will suggest the most likely tables. The following derived tables will need to be configured: `base` and `ad views`.

Verify that these match the desired tables and click **Save Settings**. If the tables have the required columns, you can navigate to the dashboard pages in the sidebar and explore your data.

### Explore the dashboards

You can browse the following pages:

1. **Overview**: key metrics across all media, including number of plays, average play time, and number of unique viewers. It also shows a table with details and metrics for all media items.
2. **Content**: choose one piece of media to see metrics such as total plays and total watch time, as well as charts for plays over time by platform, an engagement funnel, and retention across the media playtime.
3. **In-Media Ads**: ads engagement, including impressions over time, clicks, and click-through rate.

### Choose the date range and filters

At the top of each page, there is an option to choose the date range. The **Content** and **In-Media Ads** pages also have filters for individual items of media. These options let you narrow down the data shown in the dashboard to what you are interested in.

---

# Set up classic alerts
> Set up cost-effective classic alerts for new failed events and daily digests using internal telemetry data.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/classic-alerts/

Snowplow can send two types of pipeline alerts to help you monitor failed events:

- **New failed event:** receive an alert within 10 minutes of a new type of event failure being detected on your pipeline.
- **Failed event digest**: receive a daily digest of all failed event activity in the previous 48-hour period.

To receive alerts you must have the failed events [monitoring](/docs/monitoring/) feature switched on in Snowplow Console.

## Subscribing to alerts

- Login to Snowplow Console
- Locate the pipeline you wish to set up alerts for in the left-hand navigation
- Click on the `Configuration` tab, then the `Pipeline alerts` section

![Snowplow Console pipeline Configuration tab showing the Pipeline alerts section with two alert types listed: \&quot;Failed Events (digest)\&quot; and \&quot;New Failed Event\&quot;, each showing \&quot;No recipients currently set to receive these alerts\&quot; and a Manage link.](/assets/images/image-febe5bcb4732a53fa5a1b94da9e47175.png)

- Click `Manage` for the alert you wish to subscribe to
- Add one or more email addresses by typing them into the input and clicking `Add recipient`
- Once you have added all recipients, click `Save Changes`

![Manage Notification modal for \&quot;Failed events (digest)\&quot; showing an email address input field, an Add recipient button, a current recipient (joe@example.com), and Save Changes and Cancel buttons.](/assets/images/image-1-fb3fab59f7b5cc71a4e25c13ff7e4c21.png)

---

# Create failed event alerts in Console
> Configure email or Slack alerts for failed events with custom filters and thresholds in the data quality dashboard.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/failed-event-alerts/creating-alerts/

Set up failed event alerts to receive notifications when failed events occur in your data pipeline.

You'll need access to the [data quality dashboard](/docs/monitoring/#data-quality-dashboard).

To create an alert, go to Snowplow Console:

1. Navigate to **Data Quality** in the left sidebar
2. Click **Manage alerts** in the top-right corner
3. Click **Create alert**

![Manage alerts page in an empty state showing \&quot;No alerts added\&quot; and a Create alert button](/assets/images/data_quality_create_alert-afeeedce7f41da59235c86224c07e133.png)

## Configure destination

Choose how you want to receive notifications:

### Email notifications

1. Select **Email** as destination
2. Add recipient email addresses
3. Click **Add filters** to configure filters
4. Configure triggers as needed
5. Enter alert name (e.g., "mobile-app")

![Create alert form with Email selected as the destination, two recipient addresses added, trigger set to \&quot;When above value\&quot; with insights showing 750 failed events per day and a suggested threshold of 1.50k, delivery set to Daily, and the alert named \&quot;web-login\&quot;](/assets/images/data_quality_create_email_alert-d7a41f749266f8b2922ae2bf52587159.png)

### Slack notifications

1. Select **Slack** as destination
2. Select Slack channel from dropdown
3. Click **Add filters** to configure filters
4. Configure triggers as needed
5. Enter alert name (e.g., "web-app")

![Create alert form with Slack selected as the destination and #slack-alerts-test chosen as the Slack channel](/assets/images/data_quality_create_slack_alert-5fb2a749236fc0c5ee189561f1518d36.png)

When no active Slack integration is found, a `Connect with Slack` button will appear instead of the list of channels.

![Destination section showing a \&quot;Connect with Slack\&quot; button, which appears when no active Slack integration is found](/assets/images/data_quality_connect_slack-19cd34be33c56ff687d46d51edee16a4.png)

A Slack consent screen will appear.

![Slack permission screen requesting access for \&quot;Snowplow Notifications\&quot; to view channel content and perform actions in the Snowplow Slack workspace](/assets/images/data_quality_slack-619e081be9bee27143d1b537cda3ec1d.png)

To select channels in the UI, first add the app to those channels. In Slack:

1. Open the channel where you want notifications
2. Type `@Snowplow Notifications` and send
3. Click "Add them" when prompted

![Slackbot message saying \&quot;You mentioned @Snowplow Notifications, but they\&#39;re not in this channel,\&quot; with Add them and Do nothing buttons](/assets/images/data_quality_slack_invite-9dfb992f8c26e95209f7343bf6f215f0.png)

Once a Slack alert is configured you will see a confirmation notification in the selected Slack channel.

![Slack message from Snowplow Notifications confirming a successful integration, showing alert name \&quot;web-login\&quot; and active filters for ResolutionError and ValidationError issue types on the mobile-login data structure](/assets/images/data_quality_slack_confirmation-6bc7271ab78f4dd95a12da4fc7251319.png)

## Set up filters

Configure when alerts should trigger:

1. **Issue types**: select `ValidationError`, `ResolutionError`, or both
2. **Data structures**: choose specific data structures (all versions will apply)
3. **App IDs**: filter by application identifiers

![Filters panel showing ResolutionError and ValidationError selected as issue types, the mobile-login data structure filtered, and no app IDs configured](/assets/images/data_quality_filters-3a3e756dad9c7851f98b9e71d6f72796.png)

## Configure triggers

Set up when alerts should be triggered based on failed event conditions:

### Trigger types

Choose from the available trigger options:

- **When above value**: Set an absolute threshold for failed events (e.g., 1,500 failed events per hour)
- **On any issue**: Alert when any failed events are detected

![Alert trigger type selector with two options: \&quot;When above value\&quot; for setting an absolute failed event threshold, and \&quot;On any issue\&quot; for alerting whenever any failed events are detected](/assets/images/data_quality_trigger_types-a758467a4f7b9abd6106b58ebb2bb4ce.png)

### Threshold configuration

When using "When above value" trigger:

1. **Value**: Enter the threshold number of failed events
2. **Time period**: Select the time window (10 minutes, hour, or day)
3. **Deliver**: Choose notification frequency (daily, weekly, or monthly)

![Alert threshold configuration panel showing Value, Time period, and Deliver fields, alongside an Insights panel with average event volume, a suggested threshold value, and an Apply suggestion button based on the last 7 days of data](/assets/images/data_quality_absolute_threshold-9ac189a9c8981ce3e989281a77e5ed9a.png)

#### Insights from recent data

When configuring a threshold alert, the Insights panel displays data from the last 7 days to help you choose an appropriate threshold:

- **Average events volume** - Shows the average counts of valid and failed events (per relevant time unit)
- **Suggested threshold** - Provides a recommended threshold value based on recent patterns
- **Apply suggestion** - Click to automatically populate the threshold value field with the suggested value

Use these insights to set realistic thresholds that reflect your actual event patterns.

**Alert delivery:** If the trigger condition is met, the alert is sent once per delivery interval you selected.

## Complete setup

1. Review your configuration
2. Click **Confirm** to create the alert
3. Your alert will appear in the alerts list

## Alert frequency

Alerts are checked every 10 minutes. You'll receive notifications when new failed events match your filter criteria.

---

# Failed event alerts
> Automatically notify via email or Slack when validation errors, resolution errors, or data quality issues occur in your pipeline.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/failed-event-alerts/

Failed event alerts automatically notify you when [failed events](/docs/fundamentals/failed-events/) occur in your data pipeline. Set up alerts to receive notifications via email or Slack when validation errors, resolution errors, or other data quality issues arise.

- The alerting system monitors your failed events and sends notifications based on the filters you configure.
- Alerts are checked every 10 minutes and sent to your specified destinations when matching failed events are detected.

These alert destinations are supported:

- **Email**: send notifications to one or more email addresses
- **Slack**: send notifications to specific Slack channels

## What you can filter on

Configure alerts to trigger only for specific types of failed events:

- **Issue types**: select `ValidationError`, `ResolutionError`, or both
- **Data structures**: filter by specific schemas or event types
- **App IDs**: filter by application identifiers

## Getting started

1. Navigate to the data quality dashboard
2. View your failed events overview
3. Click **[Manage](/docs/monitoring/alerts/failed-event-alerts/managing-alerts/)** alerts to set up notifications
4. **[Create](/docs/monitoring/alerts/failed-event-alerts/creating-alerts/)** and configure your first alert

![Data quality dashboard header for Pipeline 3 showing the dashboard description and a \&quot;Manage alerts\&quot; button](/assets/images/dq_manage_alerts_button-6db5c87cc08cc32cdc8e7e7fac30b3b4.png)

---

# Manage failed event alerts
> Manage existing failed event alerts by editing filters, changing destinations, or deleting alerts in the data quality dashboard.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/failed-event-alerts/managing-alerts/

This page explains how to edit, delete, or review existing failed event alerts.

## View alerts

1. Navigate to **Data Quality** in the left sidebar
2. Click **Manage alerts** in the top-right corner
3. View all configured alerts with their destinations

![Manage alerts page showing two configured alerts: \&quot;web-login\&quot; via Slack and \&quot;mobile-login\&quot; via Email, with a \&quot;+ Create alert\&quot; button](/assets/images/data_quality_list_alerts-0f080579a99d99787076ffd84fbb3862.png)

## Edit an alert

1. Click the arrow next to the alert name

2. Modify destination, filters, triggers, or recipients:

   - **Destination**: Change email addresses or Slack channels
   - **Filters**: Update issue types, data structures, or App IDs
   - **Triggers**: Switch between "When above value" and "On any issue", adjust threshold values and delivery frequency

3. Click **Save** to update

## Delete an alert

1. Click the arrow next to the alert name
2. Click on the three dots button
3. Click **Delete**
4. Confirm deletion

## Alert behavior

### Trigger frequency

- **Threshold-based triggers**: Alerts are evaluated against the specified threshold and time window. Notifications are sent according to the delivery frequency (daily, weekly, or monthly) when conditions are met.
- **"On any issue" triggers**: Alerts fire immediately when any failed events are detected, subject to the configured delivery frequency.

### Multiple notifications

Alerts trigger when new failed events match your filters. You may receive multiple notifications for the same failed events in the following scenarios:

- **Rolling window detection**: alerts use rolling time windows to detect failed events. The same failed event type will continue triggering notifications as each consecutive window passes and detects the events again.
- **Overlapping alert configurations**: Multiple alerts may capture the same failed events when their filter criteria overlap. This results in duplicate notifications, especially when alerts are configured to send to the same destination (same Slack channel or email address).
- **Delivery frequency**: Alerts respect the configured delivery frequency to prevent notification spam, sending at most once per the specified period when trigger conditions are met.

---

# Set up failed event alerts
> Configure alerts for failed events using warehouse-based alerts or classic alerts with email and Slack notifications.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/

Snowplow can alert you when a new failed event has occurred. There are two different implementations available to choose from:

- [Failed event alerts](/docs/monitoring/alerts/failed-event-alerts/) run queries directly against the your data warehouse every 10 minutes. This approach keeps warehouse workers continuously active, resulting in higher compute costs compared to classic alerts, but provides faster detection, are richer alert context, and are more configurable.

- [Classic alerts](/docs/monitoring/alerts/classic-alerts/) offer cost-effective monitoring by executing queries against our internal telemetry data warehouse tables. This approach minimizes your costs since the computational load runs on our systems.

## Feature comparison

| Feature                                       | Classic alerts | Failed event alerts |
| --------------------------------------------- | -------------- | ------------------- |
| Can alert on new failed events                | ✅              | ✅                   |
| Can send a digest of failed events for a week | ✅              | ✅                   |
| Notify via Email                              | ✅              | ✅                   |
| Notify via Slack                              | ❌              | ✅                   |
| Filters                                       | ❌              | ✅                   |
| Does not affect pipeline cost                 | ✅              | ❌                   |

---

# Access raw failed events from S3 or GCS buckets
> Retrieve and query raw failed event payloads directly from S3 or GCS file storage using Athena or BigQuery.
> Source: https://docs.snowplow.io/docs/monitoring/exploring-failed-events/file-storage/

On AWS and GCP, when failed events are generated on your pipeline, the raw event payload along with details about the failure are saved into file storage (S3 on AWS, GCS on Google Cloud).

> **Info:** If you followed the [Snowplow Self-Hosted quick start guide](/docs/get-started/self-hosted/quick-start/) on GCP, you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/) to save failed events into GCS, as it’s currently not included in the Terraform scripts.

## Retrieving raw data

You can directly access and download examples of events that are failing from file storage. This is useful for further investigation, and also required to design a recovery operation.

**AWS:**

Login to your AWS Console account and navigate to the sub-account that contains your Snowplow pipeline. Then navigate to your S3 storage buckets.

You should find a bucket with a name ending in `-kinesis-s3-bad` and within that a folder with your pipeline name e.g. `prod1`.

![AWS S3 console bucket list filtered to show one result: a bucket named ending in -kinesis-s3-bad](/assets/images/failed-evs-s3-1-84031c8dbe10799d17c6f929c7d5fd54.jpg)

![AWS S3 bucket contents showing a single folder named prod1](/assets/images/failed-evs-s3-2-5d381d124833dde43bdaff289eacd9f2.jpg)

Navigate into this folder and you should see `partitioned` (search if it isn't visible).

![AWS S3 prod1 folder filtered to show a single folder named partitioned](/assets/images/failed-evs-s3-3-002938da54b3c9eb577de32ce9e4ae42.jpg)

Within this folder, there will be a subfolder for each type of failed event. Select the relevant type for the failed events you wish to find.

![AWS S3 partitioned folder containing six subfolders, one per failed event type: adapter\_failures, collector\_payload\_format\_violation, enrichment\_failures, generic\_error, schema\_violations, and tracker\_protocol\_violations](/assets/images/failed-evs-s3-4-76b13fa30ce9af8ec672e4b5e23f5a36.jpg)

You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.

![AWS S3 folder listing showing individual failed event files in gz format with timestamps and file sizes](/assets/images/failed-evs-s3-5-08e23c59d143d1cb880d89e84e16a35b.jpg)

**GCP:**

Login to your Google Cloud Platform account and navigate to the project that contains your Snowplow pipeline. Then navigate to your Google Cloud Storage buckets.

You should find a bucket named with a prefix of `sp-storage-loader-bad`.

![Google Cloud Storage browser showing one bucket named with a sp-storage-loader-bad prefix, located in europe-west1](/assets/images/failed-evs-gcs-1-ba52c0737b95e4a2f54777cfccef3df2.jpg)

Navigate into this folder and you should see `partitioned` (search if it isn't visible).

![Google Cloud Storage bucket contents showing three folders: 2019, 2020, and partitioned](/assets/images/failed-evs-gcs-2-a91853d0d0df4501d25a7f4d3a964281.jpg)

Within this folder, there will be a subfolder for each type of failed event. Select the relevant type for the failed events you wish to find.

![Google Cloud Storage partitioned folder containing six subfolders, one per failed event type: adapter\_failures, collector\_payload\_format\_violation, enrichment\_failures, generic\_error, schema\_violations, and tracker\_protocol\_violations](/assets/images/failed-evs-gcs-3-46e49ccef31512c9375deafa5f115c96.jpg)

You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.

![Google Cloud Storage folder showing two year-based subfolders: 2020 and 2021](/assets/images/failed-evs-gcs-5-c30a145ebcd28f00ab3afdb08e2a7a2f.jpg)

![Google Cloud Storage folder showing month-based subfolders numbered 01 through 10](/assets/images/failed-evs-gcs-6-25555744ddb50b7026df16975f0350f6.jpg)

![Google Cloud Storage folder listing showing individual failed event output files from October 2021 with sizes ranging from approximately 10 to 120 KB](/assets/images/failed-evs-gcs-7-26a0e6a98aea028185da83ff11409ca9.jpg)

***

## Using Athena or BigQuery

[Athena](https://aws.amazon.com/athena/) on AWS and [BigQuery](https://cloud.google.com/bigquery) on GCP are tools that let you query your failed events, using the cloud storage files as a back-end data source.

```sql
SELECT data.failure.messages FROM adapter_failures

WHERE from_iso8601_timestamp(data.failure.timestamp) > timestamp '2020-04-01'
```

This approach is handy for debugging your pipeline without the need to load your failed events into a separate database.

Before you can query this data, you need to create corresponding tables in Athena or BigQuery as we explain below. Each different failed event type (e.g. schema violations, adapter failures) has a different schema, so you will need one table per event type.

## Creating the tables

**AWS:**

Go to [the Athena dashboard](https://eu-central-1.console.aws.amazon.com/athena/home) and use the query editor. Start by creating a database (replace `{{ DATABASE }}` with the name of your pipeline, e.g. `prod1` or `qa1`):

```sql
CREATE DATABASE IF NOT EXISTS {{ DATABASE }}
```

Then run each SQL statement provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/athena) by copying them into the Athena query editor. We recommend creating all tables, although you can skip the ones you are not interested in.

> **Info:** Note that the SQL statements contain a few placeholders which you will need to edit before you can create the tables:
>
> - `{{ DATABASE }}` — as above, change this to the name of your pipeline, e.g. `prod1` or `qa1`.
> - `s3://{{ BUCKET }}/{{ PIPELINE }}` — this should point to the directory in S3 where your bad rows files are stored.

![AWS Athena query editor showing a CREATE EXTERNAL TABLE statement being executed, with 12 failed event tables visible in the left panel and a \&quot;Query successful\&quot; result](/assets/images/athena-create-table-9b949019dd6014dd9c28e45da9f37ba2.png)

**GCP:**

> **Info:** If you followed the [Snowplow Self-Hosted Quick Start guide](/docs/get-started/self-hosted/quick-start/), you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/) to save failed events into GCS, as it’s currently not included in the Terraform scripts.

> **Note:** These instructions make use of the [`bq` command-line tool](https://cloud.google.com/bigquery/docs/bq-command-line-tool) which is packaged with the [Google cloud SDK](https://cloud.google.com/sdk/docs). Follow the SDK instructions for how to [initialize and authenticate the SDK](https://cloud.google.com/sdk/docs/initializing). Also take a look at the [BigQuery dashboard](https://console.cloud.google.com/bigquery) as you run these commands, so you can see your tables as you create them.

Create a dataset to contain your failed event tables:

```bash
bq mk --data_location=EU bad_rows_prod1

# Dataset 'my-snowplow-project:bad_rows_prod1' successfully created.
```

The `--data-location` should match the location of your bad rows bucket. Also replace `prod1` with the name of your pipeline.

Next, download the table definitions provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery) in JSON format.

> **Info:** Each table definition contains a `{{ BUCKET }}` placeholder which needs to be changed to the GCS bucket where your bad rows files are stored (e.g. `sp-storage-loader-bad-prod1-com_acme`).

Now run `bq mk` for each table definition in turn. Use the `--external_table_definition` parameter so that BigQuery uses the bucket as the back-end data source. Here is how to run the command for the first three tables (note that you should change the dataset name `bad_rows_prod1` to match the dataset you just created):

```bash
bq mk \

  --display_name="Adapter failures" \

  --external_table_definition=./adapter_failures.json \

  bad_rows_prod1.adapter_failures



# Table 'my-snowplow-project:bad_rows_prod1.adapter_failures' successfully created.



bq mk \

  --display_name "Schema violations" \

  --external_table_definition=./schema_violations.json \

  bad_rows_prod1.schema_violations



# Table 'my-snowplow-project:bad_rows_prod1.schema_violations' successfully created.



bq mk \

  --display_name "Tracker protocol violations" \

  --external_table_definition=./tracker_protocol_violations.json \

  bad_rows_prod1.tracker_protocol_violations



# Table 'my-snowplow-project:bad_rows_prod1.tracker_protocol_violations' successfully created.
```

Run the corresponding commands for the remaining table definitions. We recommend creating all tables, although you can skip the ones you are not interested in.

> **Tip:** BigQuery has an “Auto-detect” feature to automatically generate the table definition for you by inspecting the file contents. So you might wonder why it is necessary to provide explicit schema definitions for your tables.
>
> There are two potential pitfalls when using the autogenerated schema with the Snowplow bad rows files:
>
> - _Optional fields_. BigQuery might not “notice” that a field exists, depending on the sample of data used to detect the schema.
> - _Polymorphic fields_, e.g. `error` that can be either a string or an object. BigQuery will throw an exception if it sees an unexpected value for a field. Our table definitions use the `JSON` data type for these fields.

***

## Querying the data

**AWS:**

As example of using your Athena tables, you might start by getting counts of each failed event type from the last week. Repeat this query for each table you have created:

```sql
SELECT COUNT(*) FROM schema_violations

WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
```

![AWS Athena query editor showing a SELECT COUNT(\*) query on the schema\_violations table filtered to the last 7 days, returning a count of 3187](/assets/images/athena-count-3c29355bdfad7e01cc24cb3bd09105b9.png)

If you have schema violations, you might want to find which tracker sent the event:

```sql
SELECT data.payload.enriched.app_id, COUNT(*) FROM schema_violations

WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())

GROUP BY data.payload.enriched.app_id
```

You can do a deeper dive into the error messages to get a explanation of the last 10 failures:

```sql
SELECT message.field AS field,

       message.value AS value,

       message.error AS error,

       message.json AS json,

       message.schemaKey AS schemaKey,

       message.schemaCriterion AS schemaCriterion

FROM schema_violations

CROSS JOIN UNNEST(data.failure.messages) AS t(message)

ORDER BY data.failure.timestamp DESC

LIMIT 10
```

**GCP:**

You can query your tables from the query editor in the [BigQuery console](https://console.cloud.google.com/bigquery). You might want to start by getting counts of each failed event type from the last week. This query will work, but it is relatively expensive because it will scan all files in the `schema_violations` directory:

```sql
SELECT COUNT(*) FROM bad_rows_prod1.schema_violations

WHERE data.failure.timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY);
```

You can construct a more economical query by using the `_FILE_NAME` pseudo column to restrict the scan to files from the last week:

```sql
SELECT COUNT(*) FROM bad_rows_prod1.schema_violations

WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY);
```

You can repeat that query for each table you created in your bad rows dataset.

![Google BigQuery query editor showing a SELECT COUNT(\*) query on the schema\_violations table filtered to the last 7 days using the \_FILE\_NAME column, returning a count of 16111](/assets/images/bigquery-count-06422ef48c38bef0a72bc39b1b739bf9.png)

If you have schema violations, you might want to find which tracker sent the event:

```sql
SELECT data.payload.enriched.app_id, COUNT(*) FROM bad_rows_prod1.schema_violations

WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)

GROUP BY data.payload.enriched.app_id;
```

If you have tracker protocol failures, you can do a deeper dive into the error messages to get a explanation of the last 10 failures:

```sql
SELECT message.field AS field,

       message.value AS value,

       message.error AS error,

       message.expectation AS expectation,

       message.schemaKey AS schemaKey,

       message.schemaCriterion AS schemaCriterion

FROM bad_rows_prod1.tracker_protocol_violations,

UNNEST(data.failure.messages) AS message

WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)

ORDER BY data.failure.timestamp DESC

LIMIT 10;
```

**Digging deeper**

You might notice that the `error` field in the result of the query above has the `JSON` type. This is because depending on the variety of the failed event, the `error` might be a simple string or a complex object with additional detail.

For example, the “invalid JSON” message might have this `error`:

```json
"invalid json: expected false got 'foo' (line 1, column 1)"
```

In contrast, in case of a failure to resolve Iglu server, the value in the `error` field would look like this, with “sub-errors” inside:

```json
{

  "error": "ResolutionError",

  "lookupHistory": [

    {

      "attempts": 1,

      "errors": [

        {

          "error": "RepoFailure",

          "message": "Unexpected exception fetching: org.http4s.client.UnexpectedStatus: unexpected HTTP status: 404 Not Found"

        }

      ],

      "lastAttempt": "2021-10-16T17:20:52.626Z",

      "repository": "Iglu Central"

    },

    ...

  ]

}
```

You can figure out what to expect from such a field by looking at the JSON schema for the respective type of failed events, in this case the [tracker protocol violations schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema/1-0-0). The mapping between the various failed event tables and the corresponding JSON schemas is [here](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery).

BigQuery has a variety of JSON functions that allow you to extract data from within complex objects. For instance, if you are interested in Iglu repositories that failed to resolve, you can use something like this:

```sql
SELECT DISTINCT(JSON_VALUE(message.error.lookupHistory[0].repository))

FROM ...

WHERE ...

AND message.error.lookupHistory IS NOT NULL
```

It’s also possible, although unwieldy, to reduce all `error`s to a single string:

```sql
-- Unnest individual messages for each failed event

WITH unnested_messages AS (

  SELECT message, CASE

    -- resolution errors

    WHEN message.error.lookupHistory IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.lookupHistory[0].errors)

    -- event validation errors

    WHEN message.error.dataReports IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.dataReports)

    -- schema validation errors

    WHEN message.error.schemaIssues IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.schemaIssues)

    -- other errors

    ELSE [TO_JSON(STRUCT(message.error as message))]

  END AS errors

FROM bad_rows_prod1.tracker_protocol_violations,

UNNEST(data.failure.messages) AS message

WHERE ...)



SELECT JSON_VALUE(error.message) AS error

FROM unnested_messages,

UNNEST(errors) AS error
```

In the future, we plan to simplify the schemas of failed events so that they are more uniform and straightforward to query.

***

---

# Load failed events into warehouse or lake
> Load common types of failed events to a separate table in your warehouse or lake to analyze them easily.
> Source: https://docs.snowplow.io/docs/monitoring/exploring-failed-events/

> **Note:** This feature is available since Enrich 5.0.0 and works with:
>
> - Snowflake Streaming Loader
> - BigQuery Loader (since version 2)
> - Lake Loader
> - Databricks Streaming Loader

For the common failures (validation and enrichment), you can configure continuous loading of any offending events into _a separate table_ in your warehouse or lake. This way, you can easily inspect them and decide how they might be patched up (e.g. with SQL) and merged with the rest of your data.

> **Note:** This feature is not retroactive, i.e. only failed events that occur _after it’s enabled_ will be loaded into your desired destination.

## Format

The format of the failed events loaded into your warehouse or lake is [the same as for your atomic events](/docs/fundamentals/canonical-event/). All the standard fields are present (unless themselves invalid — see below). This allows you to query and aggregate this data easily, e.g. if you want to see the number of failures per `app_id`.

There are two differences compared to regular events.

**Invalid data is removed from the event.** This principle applies to all columns:

- Any invalid standard column (e.g. `geo_country`) will be set to `null` instead of the offending value.
- Likewise, any column containing the JSON for a self-describing event (`unstruct_...`) will be set to `null` if that JSON fails validation.
- Finally, for entity columns (`contexts_`), if one entity is invalid, it will be removed from the array of entities. If all entities are invalid, the whole column will be set to `null`.

For more information about the different columns in Snowplow data, see [how Snowplow data is stored in the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

**There is an extra column with failure details.** The column is named `contexts_com_snowplowanalytics_snowplow_failure_1`. In most cases, it will also contain the invalid data in some form. See the [next section](#example-failed-event) for an example.

## Example failed event

Here is an example of what the `contexts_com_snowplowanalytics_snowplow_failure_1` column might contain. Note that a single failed event might have more than one error. In this case, there is a required field missing, and also an unrecognized field present.

```js
[

  {

    // timestamp of the failure

    "timestamp": "2025-01-15T14:12:50.498148Z",



    // failure type and failure schema version

    "failureType": "ValidationError",

    "_schema_version": "1-0-0",



    // the component where the failure happened

    "componentName": "snowplow-enrich-kafka",

    "componentVersion": "5.1.2",



    // the schema of the offending event

    "schema": "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",



    // any properties which were invalid

    "data": {

      "invalidProperty": "This schema doesn't have this property"

    },



    // there can be multiple errors per event

    "errors": [

      {

        "keyword": "required",

        "message": "$.targetUrl: is missing but it is required",

        "path": "$",

        "source": "unstruct",

        "targets": [

          "targetUrl"

        ]

      },

      {

        "keyword": "additionalProperties",

        "message": "$.invalidProperty: is not defined in the schema and the schema does not allow additional properties",

        "path": "$",

        "source": "unstruct",

        "targets": [

          "invalidProperty"

        ]

      }

    ]

  }

]
```

## Configure

To use this feature, you will first need to enable the stream that contains failed events in the [Snowplow TSV format](/docs/pipeline/enriched-tsv-format/) suitable for loading into your warehouse or lake.

The instructions below are for Snowplow CDI customers. For Snowplow Self-Hosted, you will need to configure this manually via Terraform.

> **Note:** An additional stream (Kinesis, Pub/Sub or Event Hubs on AWS, GCP and Azure respectively) will be reflected in your cloud infrastructure costs (unless you are using [Snowplow Cloud](/docs/get-started/#cdi-cloud)). That said, failed events are usually a tiny fraction of all events, so this stream will be minimally sized.

Open the _“Pipeline configuration”_ section for the desired pipeline and select _“Failed events stream”_.

![The Pipeline configuration page showing the \&quot;Failed events stream\&quot; option under Validation, with an Enable button to activate streaming of failed events in Snowplow TSV format](/assets/images/enable-stream-bf8168f51498a883da91d0108d639a4d.png)

Click _“Enable”_ and wait for the changes to take effect.

Now you are ready to add a loader. Click _“Add failed events loader”_, which will take you to the destinations catalog.

You can use the following loaders with the failed events stream:

- Snowflake Streaming Loader
- BigQuery Loader (since version 2)
- Lake Loader
- Databricks Streaming Loader

Pick your desired destination and follow the steps in the UI, selecting _“failed events”_ as the type of events.

![The Events type selector showing two radio button options: \&quot;Enriched events\&quot; and \&quot;Failed events\&quot;, with \&quot;Failed events\&quot; selected](/assets/images/loader-type-242d12262466aff8ea2061d94f51907f.png)

Note that as with any other loader, you will first need to create a connection to your warehouse or lake, and then the loader itself.

> **Warning:** Some of the problems that cause failed events could lead them to contain personally identifiable information (PII). For example, a validation error could stem from PII placed in the wrong field, and that field might not be pseudonymized, leaving PII exposed in the error message. Or the [PII enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) itself might have failed.
>
> For this reason, we strongly recommend loading failed events into a separate schema (in case of a warehouse) or storage location (in case of a data lake) compared to your atomic events. This allows you to restrict access to failed events.
>
> Keep this in mind when creating the connection.

---

# Monitor failed events
> Monitor and manage data quality with tools for tracking failed events, data quality dashboards, and Console API access.
> Source: https://docs.snowplow.io/docs/monitoring/

[Failed events](/docs/fundamentals/failed-events/) are events the pipeline had some problem processing. Snowplow pipelines separate events that are problematic in order to keep data quality high in downstream systems.

We provide tooling for monitoring, [loading](/docs/monitoring/exploring-failed-events/), and [recovering](/docs/monitoring/recovering-failed-events/) failed events.

Snowplow offers two different ways to monitor failed events in [Snowplow Console](https://console.snowplowanalytics.com):

- The data quality dashboard that surfaces failed events directly from your warehouse in a secure manner, making debugging easier
- The default view

You can also use the Console API to monitor failed events.

## Monitoring dashboard comparison

The data quality dashboard and the default view are both useful in monitoring for failed events. Which one to choose depends on your requirements. This table summarises the differences:

| Feature               | Data quality dashboard                | Default view                                 |
| --------------------- | ------------------------------------- | -------------------------------------------- |
| **Requirements**      | Deployed Failed Events Loader         | No additional infrastructure needed          |
| **Detail provided**   | Complete failed events JSON           | Aggregated data with redacted error messages |
| **PII security**      | Data never leaves your infrastructure | Redacted data flows through Snowplow systems |
| **Warehouse support** | Snowflake and BigQuery only           | All warehouses                               |

## Data quality dashboard

The data quality dashboard allows you to see failed events directly from your warehouse. Your browser connects directly to an API running within your infrastructure, such that no failed event information flows through Snowplow. The discussion of architecture is important as it highlights the trade-offs between the two ways of monitoring failed events. The aforementioned API is a simple proxy that connects to your warehouse and serves the failed events to your browser. The connection is fully secure, using an encrypted channel (HTTPS) and authenticating/authorizing via the same mechanism used by Console.

In order for you to be able to deploy and use the data quality dashboard, you need to sink failed events to the warehouse via a [Failed Events Loader](/docs/monitoring/exploring-failed-events/#configure). The data quality dashboard currently supports Snowflake and Bigquery connections.

![Architecture diagram showing user-generated events flowing through the pipeline to a warehouse, with the Data Quality API connecting directly to the warehouse, and the Console UI in the browser sending failed event queries to the Data Quality API while authenticating via the Console API.](/assets/images/dqd-architecture-40884e74674c075988a723be4d45b4e9.png)

In this setup, you expose an additional interface (Data Quality API) to the public internet, and all failed events information is served via that interface.

The user experience of the data quality dashboard is similar to the default view, but offers substantially more information to support resolution of tracking issues. The overview page looks as follows:

![Data quality dashboard overview showing 15.29k total data volume (10.55k valid events, 4.73k failed events), a stacked bar chart of failed event volumes by day, and a table listing ValidationError issues by data structure, app ID, app version, first and last seen dates, volume, and trend.](/assets/images/dqd-overview-5f84e3168038e22581b570180c86214e.png)

As in the default view, it is possible to change the time horizon from the 7-day default, to the last hour, last day, or last 30 days. You get a corresponding overview of event volumes, both the successful and the failed ones, for the whole time period but also split per day/hour/minute in the bar chart right below.

Complementing the graphical overview, there is a table that lists all errors (at this moment only validation and resolution errors are supported). You can quickly see, for each type of failed event, a short description of the root cause, the offending data structure, first and last seen timestamps, and the total volume.

Clicking on a particular error type will take you to a detailed view:

![Issue details page for a ValidationError showing the error message \&quot;3 classes (array, null, string) is required\&quot;, the offending data structure and app IDs, and a sample data table with columns including COLLECTOR\_TSTAMP and CONTEXTS\_COM\_SNOWPLOWANALYTICS\_SNOWPLOW\_FAILURE\_1 containing JSON failure context.](/assets/images/dqd-details-845806f880ee86b5a6418a243e41a9e7.png)

The detailed view also shows a description of the root cause and the application version ([web](/docs/events/ootb-data/app-information/#entity-definitions), [mobile](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/)). It provides a sample of the failed events in their entirety, as found in your warehouse.

Some columns are too wide to fit in the table: click on them to see the full pretty-printed, syntax-highlighted content. The most useful column to explore is probably `CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_FAILURE_1`, which contains the actual error information encoded as a JSON object:

![Value details modal displaying the full content of a CONTEXTS\_COM\_SNOWPLOWANALYTICS\_SNOWPLOW\_FAILURE\_1 column as syntax-highlighted JSON, showing schema version 1-0-0, componentName snowplow-enrich-kinesis, and lookupHistory errors with NotFound responses from Iglu Central.](/assets/images/cell-content-444ee01b40d0096460610ead840479f8.png)

Finally, you can click on the **View SQL query** button to see the SQL query that was used to fetch the failed events from your warehouse:

![View SQL query modal showing the SELECT statement used to fetch failed events from atomic\_failed.events, including lateral flatten operations on the failure context column and a WHERE clause filtering by error hash and load timestamp range.](/assets/images/sql-query-b5c2f610878be42fa64bc59788c373df.png)

### Missing warehouse permissions

When deploying a loader with the data quality add-on (API), you may encounter permission errors that prevent the dashboard from querying your warehouse.

**BigQuery:**

If your service account lacks the required [permission](https://cloud.google.com/iam/docs/service-accounts) to create BigQuery jobs, you may receive the following error:

- Error code: `21xxx`
- Description: `Missing permission 'bigquery.jobs.create' on Bigquery...`

Check if your service account has the [required role](https://cloud.google.com/bigquery/docs/access-control#bigquery):

```bash
gcloud projects get-iam-policy <PROJECT_ID> \

  --flatten="bindings[].members" \

  --filter="bindings.members:<SERVICE_ACCOUNT_EMAIL>" \

  --format="table(bindings.role)"
```

To fix the error, grant the required role to your service account (recommended):

```bash
gcloud projects add-iam-policy-binding <PROJECT_ID> \

  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \

  --role="roles/bigquery.jobUser"
```

Alternatively, if you need more granular control, create a custom role with only the `bigquery.jobs.create` permission:

```bash
gcloud iam roles create customBigQueryJobCreator \

  --project=<PROJECT_ID> \

  --title="BigQuery Job Creator" \

  --description="Create BigQuery jobs for Data Quality Dashboard" \

  --permissions="bigquery.jobs.create"



gcloud projects add-iam-policy-binding <PROJECT_ID> \

  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \

  --role="projects/<PROJECT_ID>/roles/customBigQueryJobCreator"
```

**Snowflake:**

If your role lacks `USAGE` [privilege](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#warehouse-privileges) on the active warehouse, queries cannot be performed. You may receive the following error:

- Error code: `11xxx`
- Description: `Missing required privileges on Snowflake: No active warehouse selected in the current session...`

Verify current warehouse privileges for your role:

```sql
SHOW GRANTS ON WAREHOUSE <WAREHOUSE_NAME>;

SHOW GRANTS TO ROLE <ROLE_NAME>;
```

If necessary, [grant](https://docs.snowflake.com/en/user-guide/security-access-control-overview) the `USAGE` privilege on the warehouse:

```sql
GRANT USAGE ON WAREHOUSE <WAREHOUSE_NAME> TO ROLE <ROLE_NAME>;
```

Ensure the grant is properly applied:

```sql
-- Verify the grant

SHOW GRANTS ON WAREHOUSE <WAREHOUSE_NAME>;
```

***

### Query timeouts

Long-running queries, a large volume of failed events, or resource pool exhaustion can cause the data quality dashboard to time out when fetching failed events. You may receive the following errors:

- Error codes: `12xxx` or `22xxx`
- Description: `Query exceeded timeout` or `Query execution time limit exceeded`

**BigQuery:**

Diagnose the cause of the error by running:

```sql
-- Check recent query performance

SELECT

  job_id,

  user_email,

  total_slot_ms,

  total_bytes_processed,

  TIMESTAMP_DIFF(end_time, start_time, SECOND) as duration_seconds

FROM `<PROJECT_ID>.region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT`

WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)

  AND state = 'DONE'

  AND statement_type = 'SELECT'

ORDER BY total_slot_ms DESC

LIMIT 10;
```

**Snowflake:**

Diagnose the cause of the error by running:

```sql
-- Check query history

SELECT

  query_id,

  query_text,

  warehouse_name,

  execution_time,

  queued_overload_time,

  bytes_scanned

FROM table(information_schema.query_history())

WHERE start_time >= DATEADD('day', -1, CURRENT_TIMESTAMP())

  AND execution_status = 'SUCCESS'

ORDER BY execution_time DESC

LIMIT 10;
```

***

To fix these errors, try:

- Reducing query scope

  - For Console, try switching to using legacy failed events based on telemetry data
  - If using the API, try using smaller time windows, e.g., "Last hour" or "Last day" instead of "Last 30 days"
  - Query specific error types or schemas rather than all failed events

- Optimizing warehouse performance

  - Review your warehouse configuration and query patterns
  - Consider implementing partitioning, clustering, or other optimization strategies
  - Monitor resource usage, and adjust warehouse size as needed

## Default view

The default view is a relatively simple interface that shows the number of failed events over time alongside a coarse-grained description of the problem at hand. In certain cases, the error message can be somewhat cryptic in terms of diagnosing the root cause of the error. The reason is that this information flows through Snowplow infrastructure, therefore we have redacted it substantially before it leaves your pipeline, to ensure that PII does not traverse Snowplow systems. The aggregates are served by the Console's backend:

![Architecture diagram showing user-generated events flowing through the pipeline, which contains a Failed Events Aggregator, to the warehouse. Redacted aggregates pass from the aggregator through Snowplow Infrastructure and the Console API to the Console UI in the browser.](/assets/images/aggregator-architecture-cf7c92f332ecd3224d85b0f12ac07087.png)

In this setup, you expose no additional interface to the public internet, and all failed events information is served by the Console's APIs.

Below is an example view of the failed events screen in Snowplow Console:

![Tracking quality report (30-day view) showing 502.51m total events with 2.31k failed events (0.00%), a bar chart of 30-day failure trends, and a failed events by type table listing two Validation errors: load\_succeeded/2-0-0 with 1.80k failures and page\_view/1-0-0 with 335 failures.](/assets/images/image-1024x1024-55016853cb4f2d0d6dd5804da7bff6b9.png)

This interface is intended to give you a quick representation of the volume of event failures so that action can be taken. The UI focuses on schema violation and enrichment errors at present.

We've filtered on these two types of errors to reduce noise like bot traffic that causes adapter failures for instance. All failed events can be found in your S3 or GCS storage targets partitioned by type and then date/time.

At the top there is a data quality score that compares the volume of failed events to the volume that were successfully loaded into a data warehouse.

The bar chart shows how the total number of failed events varies over time, color-coding validation and enrichment errors. The time window is 7 days be default but can be extended to 14 or 30 days.

In the table, failed events are aggregated by the unique type of failure (validation, enrichment) and the offending schema.

By selecting a particular error you are able to get more detail:

![Tracking failure details page for the page\_view/1-0-0 schema showing error messages for $.documentLocationUrl and $.documentPath maxLength violations, vendor com.google.analytics.measurement-protocol, tracker com.google.analytics.measurement-protocol-v1, and a 30-day trend chart with 335 total failures most recently on 12 May.](/assets/images/image-1-1024x1009-88e9d195e3a0ea372e494280daba3bf1.png)

The detailed view shows the error message as well as other useful metadata (when available), like `app_id`, to help diagnose the source and root cause of the error; as well as a bar chart indicating how the volume of failures varies over time for this particular failed event.

## Console API

The API that powers the Console view and dashboard is publicly available, and can be invoked with a valid token to feed your own monitoring systems.

Before you can invoke the Failed Events API, you will need to [authenticate with an API key](/docs/account-management/).

A full specification of the API can be found in [our swagger docs](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml#/Metrics/getOrganizationsOrganizationidMetricsV1PipelinesPipelineidFailed-events). It is worth pointing out that, as is the case in the UI, the data returned only contains schema validation errors and enrichment failures.

---

# Recover failed events
> Fix and reprocess failed events using SQL recovery or manual recovery from buckets.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/

Event recovery allows you to fix [failed events](/docs/fundamentals/failed-events/) in some scenarios, so that you can ensure your data is complete and accurate.

A typical recovery process runs a script over a set of failed events to solve the issues (e.g. misspelled attribute name), and then attempts to re-process these events if necessary.

The best way to recover failed events is to [set up loading failed events into a separate table in your warehouse or lake](/docs/monitoring/exploring-failed-events/). [Use SQL to recover](/docs/monitoring/recovering-failed-events/recover-with-sql/) the events.

If you don't have a Failed Events Loader, you can [manually recover](/docs/monitoring/recovering-failed-events/manual/) failed events from their S3 or GCS buckets.

---

# Event recovery configuration examples
> Example recovery configurations including pass-through flows, multiple flow scenarios, and advanced multi-operation recovery patterns.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/configuration/configuration-examples/

The most basic configuration that will run for the particular failure type: `adapter-failure`, in a particular version, and mark all others as `failed` because of missing configuration mappings would look like this:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",

  "data": {

    "iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-0": [

      {

        "name": "pass-through-flow",

        "conditions": [],

        "steps": []

      }

    ]

  }

}
```

In above scenario we would be resubmitting all the received `adapter_failures` without any modifications.

## Multiple flows example

The next example shows a way of setting up multiple configuration flows for specific failure types. Configurations will be matched top to bottom and the first from the top will be chosen.

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",

  "data": {

    "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0": [

      {

        "name": "main-flow",

        "conditions": [

          {

            "op": "Test",

            "path": "$.payload.raw.vendor",

            "value": {

              "regex": "com.snowplow\\.*"

            }

          }

        ],

        "steps": [

          {

            "op": "Replace",

            "path": "$.raw.refererUri",

            "match": "(?U)^.*$",

            "value": "https://console.snplow.com/"

          }

        ]

      },

      {

        "name": "pass-through",

        "conditions": [],

        "steps": []

      }

    ]

  }

}
```

In above scenario we would only process the `1-0-0` version of `enrichment_failures`. We would be modifying those for which vendor in raw payload starts with `com.snowplow`. For those rows that match that field value, we replace the full `refererUri` contents with `https://console.snplow.com/`. All other `enrichment_failures` will be resubmitted as they were originally (pass-through scenario).

## Putting it all together

Below is an advanced example making use of most of the features and performing multiple operations on a very specific subset of failed events.

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",

  "data": {

    "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0": [

      {

        "name": "main-flow",

        "conditions": [

          {

            "op": "Test",

            "path": "$.processor.artifact",

            "value": {

              "value": "beam-enrich"

            }

          },

          {

            "op": "Test",

            "path": "$.payload.raw.vendor",

            "value": {

              "regex": "com.snowplow\\.*"

            }

          },

          {

            "op": "Test",

            "path": "$.processor.artifact",

            "value": {

              "size": {

                "eq": 11

              }

            }

          },

          {

            "op": "Test",

            "path": "$.processor.artifact",

            "value": {

              "size": {

                "gt": 3

              }

            }

          },

          {

            "op": "Test",

            "path": "$.processor.artifact",

            "value": {

              "size": {

                "lt": 30

              }

            }

          }

        ],

        "steps": [

          {

            "op": "Replace",

            "path": "$.raw.refererUri",

            "match": "(?U)^.*$",

            "value": "https://console.snplow.com/"

          },

          {

            "op": "Remove",

            "path": "$.raw.parameters.aid",

            "match": "(?U)^.*$"

          },

          {

            "op": "Replace",

            "path": "$.raw.headers",

            "match": "X-Forwarded-Proto:.*",

            "value": "X-Forwarded-Proto: http"

          },

          {

            "op": "Replace",

            "path": "$.raw.parameters.cx.data.[?(@.schema=~iglu:org.w3/PerformanceTiming/jsonschema/1-0-0)].data.loadEventEnd",

            "match": "(?U)^.*$",

            "value": "1"

          },

          {

            "op": "Cast",

            "path": "$.raw.parameters.cx.data.[?(@.schema=~iglu:org.w3/PerformanceTiming/jsonschema/1-0-0)].data.domComplete",

            "from": "Numeric",

            "to": "Boolean"

          },

          {

            "op": "Replace",

            "path": "$.raw.parameters.cx.data.[1].data.domComplete",

            "match": "false",

            "value": "true"

          },

          {

            "op": "Cast",

            "path": "$.raw.parameters.cx.data.[?(@.data.navigationStart=~([0-9]+))].data.domComplete",

            "from": "Boolean",

            "to": "Numeric"

          }

        ]

      },

      {

        "name": "impossible-flow",

        "conditions": [

          {

            "op": "Test",

            "path": "$.processor.artifact",

            "value": {

              "value": "lorem-ipsum"

            }

          }

        ],

        "steps": []

      }

    ]

  }

}
```

---

# Configure an event recovery job
> Define recovery configuration using steps for field modifications, conditions for matching events, and flows to orchestrate recovery operations.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/configuration/

Configuration for event recovery involves the use of reusable components that help map onto specific failure types. The following components are available:

### Steps

Steps are individual modifications applied to specific failed event payloads. The steps operate on specific field values and can replace, remove/nullify or cast JSON type values.

Notice

Recovery operates on an individual event's `payload` field. As an example to reach the `vendor` field that’s located within the payload, the `path` would be `$.vendor`.

Steps are constructed in JSON object format consisting of four individual parts:

- `op` – transformation operation to perform: Replace, Remove, Cast
- `path` – JSON Path to the object where the operation is meant to happen. The path can contain specific field names (ie. `vendor`) , array ids (ie. `[1]`) or filters (by regex: `$.raw.parameters.cx.data.[?(@.data.navigationStart=~([0-9]+))].data.domComplete`).
- `match` – an expression applied when replacing field’s values with new value
- `value` – a new value to be set
- `from` / `to` - used for casting types _from_ one type _to_ another

```json
// Replace

{

  "op": "Replace",

  "path": "$.raw.parameters.aid",

  "match": "(?U)^.*$",

  "value": "https://console.snplow.com/"

}



// Remove

{

  "op": "Remove",

  "path": "$.raw.parameters.aid",

  "match": "(?U)^.*$"

}



// Cast

{

  "op": "Cast",

  "path": "

$.raw.parameters.cx.data.[?(@.data.navigationStart=~([0-9]+))].data.domComplete",

  "from": "Numeric",

  "to": "Boolean"

}
```

### Conditions

Conditions are boolean expressions that operate on failed event fields in order to match on specific structure or values.

Note that conditions can be applied to arbitrary fields and therefore a condition's path scopes its entry point as the failed events “root”.

Much like steps, conditions are constructed with the following JSON object format consisting of four individual parts:

- `op` – transformation operation to perform: Replace, Remove, Cast & Test (tests that the specified value is set in the document. If the test fails, then the patch as a whole should not apply.)
- `path` – JSON Path to object, the path can contain specific field names (ie. `raw`) , array ids (ie. `[1]`) or filters (by regex: `[?(@.schema=~iglu://\\.*)]`).
- `value` – a value matcher to match against: match a regular expression, compare directly (object equality), check size for equality, less or greater than

```json
// Compare values

{

  "op": "Test",

  "path": "$.processor.artifact",

  "value": {

    "value": "beam-enrich"

  }

}



// Match against regex

{

  "op": "Test",

  "path": "$.payload.raw.vendor",

  "value": {

    "regex": "com.snowplow\\.*"

  }

}



// Compare sizes

{

  "op": "Test",

  "path": "$.processor.artifact",

  "value": {

    "size": {

      "eq": 11

    }

  }

}

{

  "op": "Test",

  "path": "$.processor.artifact",

  "value": {

    "size": {

      "gt": 3

    }

  }

}

{

  "op": "Test",

  "path": "$.processor.artifact",

  "value": {

    "size": {

      "lt": 30

    }

  }

}
```

### Flows

Flows are sequences of Steps applied one by one. Flows are mapped onto specific steps and conditions.

```json
{

  "name": "main flow",

  "conditions": [],

  "steps": []

}
```

## I/O

In principle data is sourced from bucket storage and delivered back into a collector. Recoverable and unrecoverable failed events are stored in bucket storage. For cloud-specific locations see below tables.

#### AWS

|                |                      |                       |                              |
| -------------- | -------------------- | --------------------- | ---------------------------- |
| input          | output               | failed output         | unrecoverable output         |
| S3 (`--input`) | Kinesis (`--output`) | S3 (`--failedOutput`) | S3 (`--unrecoverableOutput`) |

#### GCP

|                 |                     |                        |                               |
| --------------- | ------------------- | ---------------------- | ----------------------------- |
| input           | output              | failed output          | unrecoverable output          |
| GCS (`--input`) | PubSub (`--output`) | GCS (`--failedOutput`) | GCS (`--unrecoverableOutput`) |

---

# Extend and customize an event recovery job
> Add custom recovery steps, conditions, and bad row type support by extending the event recovery application.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/extending/

There are several extension points for this application:

- Steps
- Conditions
- Bad Row types

## Steps

By definition steps allow performing modifications on existing bad row data points' payloads. We defined a configuration DSL for Steps that is turned into actions with `Inspectable` definitions. `Inspectable`s are data structures on which Steps can be performed. Therefore in order to add a new `Step`:

- a `config.Step` DSL structure for it has to be defined
- an implementation of action defined as per DSL has to be defined for `Inspectable`.

The latter can be described in form of `transform` function that builds upon a recursive generic JSON-transforming structure.

## Conditions

Conditions are a lot like steps but as they are triggered earlier before `Inspectable`s are created. They do operate upon JSON data and need to implement behavior for basic JSON types. Therefore in order to add a new `Condition`:

- a `config.Condition` DSL structure for it has to be defined
- functions for performing `Condition` application for basic/composite JSON types have to be defined

## Bad Row types

Once new recoverable Bad Row types are added they need to be turned into `Recoverable` by supplying appropriate `Recoverable` instances. If a new `Payload` format is added it has to be turned into `Inspectable` as well.

It is important to note that we only represent recoverable bad rows as `Recoverable` instances. Not all `BadRow`s are recoverable and we strongly believe in not representing invalid states as possible.

---

# Introduction to manual failed event recovery
> Understand the recovery workflow, prerequisites, and configuration requirements for manually recovering failed events from storage.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/getting-started/

Event recovery at its core, is the ability to fix events that have failed and replay them through your pipeline.

After inspecting failed events either in [Snowplow Console](/docs/monitoring/), or in the [partitioned failure buckets](/docs/monitoring/exploring-failed-events/file-storage/), you can determine which events are possible to recover based on what the fix entails.

With recovery it is possible to:

- replace values - e.g. correct a typo in a schema name for validation
- remove values - e.g. remove improperly encoded values from a URL string
- cast JSON types - e.g. change a property's type from `string` to `integer`

If your failed events would not be fixed by applying the above, they currently would be considered unrecoverable. Due to the fact that there might be a mix of recoverable and unrecoverable data in your storage, event recovery uses configuration in order to process only a subset of the failed events.

### What you'll need to get started

The typical flow for recovery and some prerequisites to consider would be: **Understanding the failure issue**

- Familiarity with the [failed event types](/docs/fundamentals/failed-events/)
- Access to S3 or GCS buckets with failed events

**Configuring a recovery**

- Understanding the [configuration structure](/docs/monitoring/recovering-failed-events/manual/configuration/)
- Comfort with using Regular Expressions (RegEx)

**Testing the configuration**

- Ability to edit/run a Scala script locally

**Run the recovery**

- AWS sub account or GCP project admin access in order to create a recovery user

**Monitor the recovery**

- Access to DataFlow UI (GCP) or EMR reporting (AWS)

---

# Manual event recovery workflows
> Explore data structures, build configurations, validate recovery flows, and test jobs locally using scripting utilities.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/hints-workflows/

Most of the following workflows come from testing/working with elaborate bad row structures and configurations and rely heavily on the supplied scripting utility. To set it up locally follow [the testing documentation](/docs/monitoring/recovering-failed-events/manual/testing/) or starting with an empty ammonite script file (ie. `test.sc`), set up Snowplow's repository and dependencies:

```scala
interp.repositories() ++= Seq(coursierapi.MavenRepository.of("http://maven.snplow.com/releases/"))

@

import $url.{`https://raw.githubusercontent.com/snowplow-incubator/snowplow-event-recovery/feature/recovery-typeclasses/scripts/Recovery.sc` => Recovery}, Recovery._

import $ivy.`com.snowplowanalytics::snowplow-event-recovery-core:0.2.0-rc10`, com.snowplowanalytics.snowplow.event.recovery._, config._, json._
```

## Exploring data structures

The new bad row format is designed to be, readable and self-contained, however some fields are not captured in [JSON schemas for bad row format](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows).

The unspecified fields can vary from plain JSON types through JSON objects to inline JSON structures and even Base64-encoded.

To explore the data structures we are able to use the convenience scripts as described in Testing section.

Given we got a bad row JSON structure we can turn it into navigable JSON object that we can step through and encode/decode using for example:

```scala
val badrow = """{"schema":"iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0","data":{"processor":{"artifact":"beam-enrich","version":"1.0.0-rc5"},"failure":{"timestamp":"2020-02-17T09:28:18.100Z","messages":[{"enrichment":{"schemaKey":"iglu:com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-0","identifier":"api-request"},"message":{"error":"Error accessing POJO input field [user]: [java.lang.NoSuchMethodException: com.snowplowanalytics.snowplow.enrich.common.outputs.EnrichedEvent.user_id-foo()]"}}]},"payload":{"enriched":{"app_id":"console","platform":"web","etl_tstamp":"2020-02-17 09:28:18.095","collector_tstamp":"2020-02-17 09:28:16.560","dvce_created_tstamp":"2020-02-17 09:28:16.114","event":"page_view","event_id":"2dfeb9b7-5a87-4214-8a97-a8b23176856b","txn_id":null,"name_tracker":"msc-gcp-stg1","v_tracker":"js-2.10.2","v_collector":"ssc-1.0.0-rc4-googlepubsub","v_etl":"beam-enrich-1.0.0-rc5-common-1.0.0","user_id":null,"user_ipaddress":"18.194.133.57","user_fingerprint":null,"domain_userid":"d6c468de-0aed-4785-9052-b6bb77b6dddb","domain_sessionidx":13,"network_userid":"510b2f05-27e3-4fd3-b449-a2702926da5e","geo_country":"DE","geo_region":"HE","geo_city":"Frankfurt am Main","geo_zipcode":"60313","geo_latitude":50.1188,"geo_longitude":8.6843,"geo_region_name":"Hesse","ip_isp":null,"ip_organization":null,"ip_domain":null,"ip_netspeed":null,"page_url":"https://console.snowplowanalytics.com/","page_title":"Snowplow CDI","page_referrer":null,"page_urlscheme":"https","page_urlhost":"console.snowplowanalytics.com","page_urlport":443,"page_urlpath":"/","page_urlquery":null,"page_urlfragment":null,"refr_urlscheme":null,"refr_urlhost":null,"refr_urlport":0,"refr_urlpath":null,"refr_urlquery":null,"refr_urlfragment":null,"refr_medium":null,"refr_source":null,"refr_term":null,"mkt_medium":null,"mkt_source":null,"mkt_term":null,"mkt_content":null,"mkt_campaign":null,"contexts":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"39a9934a-ddd3-4581-a4ea-d0ba20e63b92\"}},{\"schema\":\"iglu:org.w3/PerformanceTiming/jsonschema/1-0-0\",\"data\":{\"navigationStart\":1581931694397,\"unloadEventStart\":1581931696046,\"unloadEventEnd\":1581931694764,\"redirectStart\":0,\"redirectEnd\":0,\"fetchStart\":1581931694397,\"domainLookupStart\":1581931694440,\"domainLookupEnd\":1581931694513,\"connectStart\":1581931694513,\"connectEnd\":1581931694665,\"secureConnectionStart\":1581931694572,\"requestStart\":1581931694665,\"responseStart\":1581931694750,\"responseEnd\":1581931694750,\"domLoading\":1581931694762,\"domInteractive\":1581931695963,\"domContentLoadedEventStart\":1581931696039,\"domContentLoadedEventEnd\":1581931696039,\"domComplete\":0,\"loadEventStart\":0,\"loadEventEnd\":0}}]}","se_category":null,"se_action":null,"se_label":null,"se_property":null,"se_value":null,"unstruct_event":null,"tr_orderid":null,"tr_affiliation":null,"tr_total":null,"tr_tax":null,"tr_shipping":null,"tr_city":null,"tr_state":null,"tr_country":null,"ti_orderid":null,"ti_sku":null,"ti_name":null,"ti_category":null,"ti_price":null,"ti_quantity":0,"pp_xoffset_min":0,"pp_xoffset_max":0,"pp_yoffset_min":0,"pp_yoffset_max":0,"useragent":"Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","br_name":null,"br_family":null,"br_version":null,"br_type":null,"br_renderengine":null,"br_lang":"en-US","br_features_pdf":0,"br_features_flash":0,"br_features_java":0,"br_features_director":0,"br_features_quicktime":0,"br_features_realplayer":0,"br_features_windowsmedia":0,"br_features_gears":0,"br_features_silverlight":0,"br_cookies":1,"br_colordepth":"24","br_viewwidth":1918,"br_viewheight":982,"os_name":null,"os_family":null,"os_manufacturer":null,"os_timezone":"Europe/Berlin","dvce_type":null,"dvce_ismobile":0,"dvce_screenwidth":1920,"dvce_screenheight":1080,"doc_charset":"UTF-8","doc_width":1918,"doc_height":982,"tr_currency":null,"tr_total_base":null,"tr_tax_base":null,"tr_shipping_base":null,"ti_currency":null,"ti_price_base":null,"base_currency":null,"geo_timezone":"Europe/Berlin","mkt_clickid":null,"mkt_network":null,"etl_tags":null,"dvce_sent_tstamp":"2020-02-17 09:28:16.507","refr_domain_userid":null,"refr_dvce_tstamp":null,"derived_contexts":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0\",\"data\":{\"useragentFamily\":\"Firefox\",\"useragentMajor\":\"72\",\"useragentMinor\":\"0\",\"useragentPatch\":null,\"useragentVersion\":\"Firefox 72.0\",\"osFamily\":\"Linux\",\"osMajor\":null,\"osMinor\":null,\"osPatch\":null,\"osPatchMinor\":null,\"osVersion\":\"Linux\",\"deviceFamily\":\"Other\"}}]}","domain_sessionid":"96958bf6-a8bf-4be8-9c67-fd957b6bc8d2","derived_tstamp":"2020-02-17 09:28:16.167","event_vendor":"com.snowplowanalytics.snowplow","event_name":"page_view","event_format":"jsonschema","event_version":"1-0-0","event_fingerprint":"5acdc8f85f9530081d1a71ec430c8756","true_tstamp":null},"raw":{"vendor":"com.snowplowanalytics.snowplow","version":"tp2","parameters":[{"name":"e","value":"pv"},{"name":"duid","value":"d6c468de-0aed-4785-9052-b6bb77b6dddb"},{"name":"vid","value":"13"},{"name":"eid","value":"2dfeb9b7-5a87-4214-8a97-a8b23176856b"},{"name":"url","value":"https://console.snowplowanalytics.com/"},{"name":"aid","value":"console"},{"name":"cx","value":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiIzOWE5OTM0YS1kZGQzLTQ1ODEtYTRlYS1kMGJhMjBlNjNiOTIifX0seyJzY2hlbWEiOiJpZ2x1Om9yZy53My9QZXJmb3JtYW5jZVRpbWluZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJuYXZpZ2F0aW9uU3RhcnQiOjE1ODE5MzE2OTQzOTcsInVubG9hZEV2ZW50U3RhcnQiOjE1ODE5MzE2OTYwNDYsInVubG9hZEV2ZW50RW5kIjoxNTgxOTMxNjk0NzY0LCJyZWRpcmVjdFN0YXJ0IjowLCJyZWRpcmVjdEVuZCI6MCwiZmV0Y2hTdGFydCI6MTU4MTkzMTY5NDM5NywiZG9tYWluTG9va3VwU3RhcnQiOjE1ODE5MzE2OTQ0NDAsImRvbWFpbkxvb2t1cEVuZCI6MTU4MTkzMTY5NDUxMywiY29ubmVjdFN0YXJ0IjoxNTgxOTMxNjk0NTEzLCJjb25uZWN0RW5kIjoxNTgxOTMxNjk0NjY1LCJzZWN1cmVDb25uZWN0aW9uU3RhcnQiOjE1ODE5MzE2OTQ1NzIsInJlcXVlc3RTdGFydCI6MTU4MTkzMTY5NDY2NSwicmVzcG9uc2VTdGFydCI6MTU4MTkzMTY5NDc1MCwicmVzcG9uc2VFbmQiOjE1ODE5MzE2OTQ3NTAsImRvbUxvYWRpbmciOjE1ODE5MzE2OTQ3NjIsImRvbUludGVyYWN0aXZlIjoxNTgxOTMxNjk1OTYzLCJkb21Db250ZW50TG9hZGVkRXZlbnRTdGFydCI6MTU4MTkzMTY5NjAzOSwiZG9tQ29udGVudExvYWRlZEV2ZW50RW5kIjoxNTgxOTMxNjk2MDM5LCJkb21Db21wbGV0ZSI6MCwibG9hZEV2ZW50U3RhcnQiOjAsImxvYWRFdmVudEVuZCI6MH19XX0"},{"name":"tna","value":"msc-gcp-stg1"},{"name":"cs","value":"UTF-8"},{"name":"cd","value":"24"},{"name":"page","value":"Snowplow CDI"},{"name":"stm","value":"1581931696507"},{"name":"tz","value":"Europe/Berlin"},{"name":"tv","value":"js-2.10.2"},{"name":"vp","value":"1918x982"},{"name":"ds","value":"1918x982"},{"name":"res","value":"1920x1080"},{"name":"cookie","value":"1"},{"name":"p","value":"web"},{"name":"dtm","value":"1581931696114"},{"name":"lang","value":"en-US"},{"name":"sid","value":"96958bf6-a8bf-4be8-9c67-fd957b6bc8d2"}],"contentType":"application/json","loaderName":"ssc-1.0.0-rc4-googlepubsub","encoding":"UTF-8","hostname":"gcp-sandbox-prod1.collector.snplow.net","timestamp":"2020-02-17T09:28:16.560Z","ipAddress":"18.194.133.57","useragent":"Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","refererUri":"https://console.snowplowanalytics.com/","headers":["Timeout-Access: ","Host: gcp-sandbox-prod1.collector.snplow.net","User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","Accept: */*","Accept-Language: en-US, en;q=0.5","Accept-Encoding: gzip, deflate, br","Origin: https://console.snowplowanalytics.com","Referer: https://console.snowplowanalytics.com/","Cookie: sp=510b2f05-27e3-4fd3-b449-a2702926da5e","X-Cloud-Trace-Context: 958285ba723e212998af29cec405e002/12535615945289151925","Via: 1.1 google","X-Forwarded-For: 18.194.133.57, 35.201.76.62","X-Forwarded-Proto: https","Connection: Keep-Alive","application/json"],"userId":"510b2f05-27e3-4fd3-b449-a2702926da5e"}}}}"""

parse(badrow).foreach(println)
```

## Building configurations

After being able to explore the data structure and figuring out what amendments to introduce, this needs to be turned into a set of individual building blocks. Configuration building blocks can be tested individually using the scripting utility. To test particular steps you can use following pattern:

```scala
operations.cast("""{"int": "1"}""", "$.int", CastType.String, CastType.Array)

operations.replace("""{"int": "1"}""", "$.int", "(?U)^.*$", "new")
```

## Validating configurations

The rules about setting up configurations are fairly strict to prevent unnecessary delay in empty job evaluations. To validate a built configuration it is easy to simply run it through a validator using the scripting utility:

```scala
val cfg = """{ "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/3-0-0", "data": { "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-*": [{"name": "passthrough", "conditions": [], "steps": []}]}}"""

configs.validate(cfg)
```

## Testing jobs locally

After validating the configuration it makes sense to try out the full configuration locally using a sample, scrambled data:

```scala
jobs.test(cfg, badrow)

jobs.testMany(cfg, List(badrow))
```

---

# Event recovery without Failed Events Loader
> Manually recover failed events from S3 or GCS buckets using low-level recovery tools for Snowplow Self-Hosted pipelines.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/

Snowplow pipelines are "non-lossy", this means if something is wrong with an event during any part of the pipeline, the event is stored in a separate storage environment rather than just discarded. See the [failed events section](/docs/fundamentals/failed-events/) for more information on the types of failures that may occur.

Besides allowing for the inspection of failed events to fix the root cause of the problem, you have the option to recover data by running a recovery process to correct an issue and "re-play" the events through your pipeline again.

Snowplow Event Recovery is available for pipelines running on AWS and GCP.

---

# Monitor event recovery jobs
> Track recovery job progress using EMR metrics, Google Dataflow UI, and business processing metrics for failed, recovered, and unrecoverable events.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/monitoring/

When running a recovery job in your production pipeline, you'll likely find it useful to keep an eye on it's progress.

In order to verify the process is running properly there are several locations in your **infrastructure** that can be monitored, depending on your runtime environment. These are: datasinks (for recovery job processed output): `failedOutput` (S3/GCS bucket), `unrecoverableOutput` (S3/GCS bucket), `output` (Kinesis/PubSub streams) and job runners (tracking job status and processing in real-time).

Beyond monitoring infrastructure, each job exposes “business” processing metrics that summarise failed, unrecoverable and recovered bad rows that have been processed.

### Amazon EMR

On EMR recovery job exposes run summary using a [built-in reporting library delivering](https://spark.apache.org/docs/latest/monitoring.html) count summaries for:

- `event-recovery.driver.summary.Recovered`
- `event-recovery.driver.summary.Failed`
- `event-recovery.driver.summary.Unrecoverable`

The metrics can be accessed trough variety of ways (sinks) and can be configured upon cluster creation parameters.

To enable desired sink set EMR’s `spark-metrics` classification parameters following [possible values](https://github.com/apache/spark/blob/master/conf/metrics.properties.template) using EMR’s classifiers.

To expose metrics over http accessible at `http://${SPARK_HOST}:4040/metrics/json`:

```json
  {

    "classification": "spark-metrics",

    "properties": {

        "spark.metrics.namespace": "event-revovery",

        "*.sink.servlet.class": "org.apache.spark.metrics.sink.MetricsServlet",

        "*.sink.servlet.period": "1",

        "*.sink.servlet.unit": "seconds",

        "*.sink.servlet.path": "/metrics/json",

        "master.sink.servlet.path": "/metrics/master/json",

        "applications.sink.servlet.path": "metrics/applications/json",

        "*.source.metrics.class": "org.apache.spark.metrics.source.Metrics"



    }
```

To push metrics to output logs to console appender:

```json
  {

    "classification": "spark-metrics",

    "properties": {

        "spark.metrics.namespace": "$name",

        "*.sink.console.class": "org.apache.spark.metrics.sink.ConsoleSink",

        "*.source.metrics.class": "org.apache.spark.metrics.source.Metrics"



    }
```

For more sinks see [Spark documentation on monitoring](https://spark.apache.org/docs/latest/monitoring.html).

### Google Dataflow

On Google Dataflow recovery job exposes realtime run metrics using a dataflow’s native functions. The metrics are:

- `recovered`
- `failed`
- `unrecoverable`

The metrics can be accessed through the web UI directly in Dataflow.

---

# Run event recovery on Flink
> Deploy event recovery jobs on Apache Flink using EMR with fault tolerance and CloudWatch metrics monitoring.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/flink/

The Flink job reads bad rows from an S3 location and stores the recovered payloads in Kinesis, unrecovered and unrecoverable in other S3 buckets.

#### Building

To build the fat jar, run:

```bash
sbt flink/assembly
```

#### Running

Event recovery jobs are usually ran using our [dataflow-runner](https://github.com/snowplow/dataflow-runner) application. A configuration template for running is [available in the repository](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner).

To run a transient EMR cluster and execute the job using the templates, download [Flink dataflow-runner templates](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner) from the repository and run:

```bash
dataflow-runner run-transient \

  --emr-playbook flink-playbook.json.tmpl \

  --emr-config flink-cluster.json.tmpl \

  --vars bucket,$BUCKET_INPUT,region,$AWS_REGION,subnet,$AWS_SUBNET,role,$AWS_IAM_ROLE,keypair,$AWS_KEYPAIR,client,$JOB_OWNER,version,$RECOVERY_VERSION,config,$RECOVERY_CONFIG,resolver,$IGLU_RESOLVER,output,$KINESIS_OUTPUT,inputdir,$BUCKET_INPUT_DIRECTORY,interval,$INTERVAL
```

Where:

- `BUCKET_INPUT` - S3 bucket containing bad events to be recovered (eg. geoffs-bad-events)
- `BUCKET_INPUT_DIRECTORY` - directory in the `BUCKET_INPUT` to use as the source for bad events
- `AWS_REGION` - region in which the job is being ran (eg. eu-central-1)
- `AWS_SUBNET` - network subnet to run the job in (eg. subnet-435010347a21886ab)
- `AWS_IAM_ROLE` - AWS IAM role to assume while running the job (eg. geoffs-recovery-role)
- `AWS_KEYPAIR` - AWS EC2 key pair to use for the instances (eg. geoffs-keypair)
- `JOB_OWNER` - tag to use to mark the owner of the job (eg. goeff)
- `RECOVERY_VERSION` - application version (eg. 0.6.0)
- `RECOVERY_CONFIG` - recovery job config as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `IGLU_RESOLVER` - iglu resolver configuration as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `KINESIS_OUTPUT` - Kinesis stream to output recovered events to
- `INTERVAL` - job checkpointing interval (we recommend starting with 10 minutes and tuning to your job's characteristics)

The job uses [checkpointing](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/checkpointing/) to allow for job restarts. Kinesis connector which is a part of the checkpointing mechanism [guarantees](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#kinesis-sinks-and-fault-tolerance) at-least-once delivery semantics and provides backpressure to the job.

#### Monitoring

EMR Flink Job emits a wide range of metrics through Cloudwatch Agent which is installed during cluster bootstrap using a provided script (see project repo's dataflow-runner directory). Custom metrics include:

- number of unrecoverable events in the run: `taskmanager_container_XXXX_XXX__Process_0_events_unrecoverable`
- number of failed recovery attempts: `taskmanager_container_XXXX_XXX__Process_0_events_failed`
- number of recovered events: `taskmanager_container_XXXX_XXX__Process_0_events_recovered`
- total number of events submitted for processing: `taskmanager_container_XXXX_XXX__Process_0_numRecordsIn`

The metrics are delivered to `snowplow/event-recovery` Cloudwatch namespace.

Given a one-second aggregation in Cloudwatch the diagram should show an always increasing metrics that should balance themselves up to total sum.

> **Note:** Flink's `numRecordsOut` metric for sinks does not reflect an actual number of records saved in the sink.

---

# Run event recovery on GCP Dataflow with Beam
> Deploy dockerized event recovery jobs on Google Cloud Dataflow using Apache Beam runtime.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/gcp-beam/

The Beam job is a dockerized purpose built Dataflow job that reads data from a GCS location specified through a pattern and stores the recovered payloads in a PubSub topic, unrecovered and unrecoverable in other GCS buckets.

## Permissions

In order to run the job a `service-account` with appropriate credentials must be set up in the project.

In order to be sure not to disrupt any service accounts that may be required for normal pipeline operations, it's recommended to create a new service account for recovery.

To create a new service account in your GCP console navigate to [IAM & Admin > Service](https://console.cloud.google.com/iam-admin/serviceaccounts) [Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts). Create a new account and grant `full project owner` permissions.

Generate a new key for the service account and save down to your local machine.

## Building

To build the docker image, run:

```bash
sbt beam/docker:publishLocal
```

## Running

To run on Apache Beam in GCP Dataflow run it through a docker-deployment the options to set are shown below.

```bash
-v {{path-to-local-key-file}}.json:/snowplow/config/credentials.json \

-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/config/credentials.json \

  snowplow/snowplow-event-recovery-beam:0.6.1 \

--runner=DataFlowRunner \

--project={{your-gcp-project-name}} \

--jobName=event-recovery-rt-pipeline \

--region={{your-project-region}} \

--gcpTempLocation=gs://sp-storage-loader-tmp-{{pipeline_tag_e.g._prod1}}-{{pipeline_name}}/temp \

--inputDirectory=gs://sp-storage-loader-bad-{{pipeline_tag_e.g._prod1}}-{{project_name}}/partitioned/** \

--outputTopic=projects/{{project_name}}/topics/{{recovery-topic}} \

--failedOutput=gs://{{create_or_choose_a_failed_output_folder}}/failed/ \

--unrecoverableOutput=gs://{{create_or_choose_an_unrecoverable_folder}}/unrecoverable/ \

--config={{base64-encoded-string of your recovery configuration}} \

--resolver={{base64-encoded-string of a schema resolver file (Iglu Central ok for default)}}
```

As this is a self-contained docker image the first flag sets the key for the deployment in the folder for use by the job.

Note that for zone GCP prefers you may need to use -a or -b zones (e.g. europe-west2-b).

Also note, that no `"` are needed for any of the strings.

---

# Run event recovery jobs
> Define and encode recovery configuration, then deploy recovery jobs using Spark, Flink, or Beam runtimes.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/

Now that we've discussed configuring the recovery, let's dive in to running it on your pipeline.

First we'll need to define configuration:

## Define configuration

The configuration consists of the `resolver-config.json` that your pipeline uses to resolve events against corresponding schema, for example:

```json
{

  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1",

  "data": {

    "cacheSize": 0,

    "repositories": [

      {

        "name": "Iglu Central",

        "priority": 0,

        "vendorPrefixes": [

          "com.snowplowanalytics"

        ],

        "connection": {

          "http": {

            "uri": "https://iglucentral.com"

          }

        }

      }

    ]

  }

}
```

Also, the`job-config.json` which describes the recovery job to be done, for example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",

  "data": {

    "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0": [

      {

        "name": "passthrough",

        "conditions": [],

        "steps": []

      }

    ]

  }

}
```

## Encode configuration

Configuration is supplied to recovery jobs as a Base64-encoded string. You can use your encoding plugin of choice, but here is an example [ammonite](http://ammonite.io/) script you can use to encode your configuration:

```scala
import java.util.Base64

import java.nio.charset.StandardCharsets

import ammonite.ops._

import $ivy.`io.circe::circe-parser:0.13.0`, io.circe._, io.circe.syntax._, io.circe.parser._



val load = (path: String)  => read! os.Path(path, base = os.pwd)

val encode = (str: String)  => Base64.getEncoder.encodeToString(str.getBytes(StandardCharsets.UTF_8))



@main def run(config: String): Unit =

  parse(load(config)) match {

    case Right(data)  => println(encode(data.asJson.noSpaces))

    case Left(err)  => println(s"Invalid JSON input in: ${err.getMessage}")

  }
```

And then just run (assuming ammonite is on your path and above script is called `encode.sc`:

```bash
amm ./encode.sc resolver-config.json

amm ./encode.sc job-config.json
```

Once the configuration files are encrypted it's time to deploy the recovery job on your pipeline.

## Deploy the job

There are several runtimes that recovery process can be deployed to:

---

# Run event recovery on Spark (legacy)
> Deploy event recovery jobs on Apache Spark using EMR and dataflow-runner.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/spark/

> **Note:** Spark module is now deprecated. Kinesis integration is not as reliable as we'd like. We suggest migrating to [Flink module](/docs/monitoring/recovering-failed-events/manual/running/flink/).

The Spark job reads bad rows from an S3 location and stores the recovered payloads in Kinesis, unrecovered and unrecoverable in other S3 buckets.

#### Building

To build the fat jar, run:

```bash
sbt spark/assembly
```

#### Running

Event recovery jobs are usually ran using our [dataflow-runner](https://github.com/snowplow/dataflow-runner) application. A configuration templates for running is [available in the repository](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner).

To run a transient EMR cluster and execute the job using the templates, download [Spark dataflow-runner templates](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner) from the repository and run:

```bash
dataflow-runner run-transient \

  --emr-playbook spark-playbook.json.tmpl \

  --emr-config spark-cluster.json.tmpl \

  --vars bucket,$BUCKET_INPUT,region,$AWS_REGION,subnet,$AWS_SUBNET,role,$AWS_IAM_ROLE,keypair,$AWS_KEYPAIR,client,$JOB_OWNER,version,$RECOVERY_VERSION,config,$RECOVERY_CONFIG,resolver,$IGLU_RESOLVER,output,$KINESIS_OUTPUT,inputdir,$BUCKET_INPUT_DIRECTORY
```

Where:

- `BUCKET_INPUT` - S3 bucket containing bad events to be recovered (eg. geoffs-bad-events)
- `BUCKET_INPUT_DIRECTORY` - directory in the `BUCKET_INPUT` to use as the source for bad events
- `AWS_REGION` - region in which the job is being ran (eg. eu-central-1)
- `AWS_SUBNET` - network subnet to run the job in (eg. subnet-435010347a21886ab)
- `AWS_IAM_ROLE` - AWS IAM role to assume while running the job (eg. geoffs-recovery-role)
- `AWS_KEYPAIR` - AWS EC2 key pair to use for the instances (eg. geoffs-keypair)
- `JOB_OWNER` - tag to use to mark the owner of the job (eg. goeff)
- `RECOVERY_VERSION` - application version (eg. 0.6.0)
- `RECOVERY_CONFIG` - recovery job config as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `IGLU_RESOLVER` - iglu resolver configuration as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `KINESIS_OUTPUT` - Kinesis stream to output recovered events to

---

# Test an event recovery configuration locally
> Validate recovery configuration and run small-scale test recoveries locally using the snowplow-event-recovery CLI tool.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/testing/

Once you know how you want to configure the recovery job, it's worthwhile to test your configuration locally, before deploying an actual recovery job.

For that purpose we provide a `snowplow-event-recovery` command line interface. Latest versions are available through [GitHub release page](https://github.com/snowplow-incubator/snowplow-event-recovery/releases/latest).

The CLI provides commands for validating (`snowplow-event-recovery validate`) and running test recoveries (`snowplow-event-recovery run`). See `snowplow-event-recovery --help` for the most up-to-date options descriptions.

### Validate

Validation checks configuration schema to verify that actual job can be ran. To validate the configuration, run:

```text
snowplow-event-recovery validate --config /opt/scenarios.json
```

Which will result in:

```text
OK! Config valid
```

Or an detailed error message:

```text
ERROR! Invalid config



Instance is not valid against its schema:

* At $.iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-*[0].conditions[0].op:

  - $.iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-*[0].conditions[0].op: does not have a value in the enumeration [Test]
```

### Test

To run a small-scale recovery test, outputting both good and failed events into files, execute:

```text
snowplow-event-recovery run --config $PWD/scenarios.json -o /tmp -i $PWD/input/ --resolver $PWD/resolver.json
```

The input folder (`--input` or `-i`) contains a line-delimited files (by default `.txt` files are parsed, but the glob extension can be configured using `--glob` or `-g` flag) minimized json failed events as seen in [example](https://github.com/snowplow-incubator/snowplow-event-recovery/tree/master/examples).

With output being saved into `/tmp` directory as a stringified payload objects into `good.txt` and json-serialized failed events into `bad.txt`.

Which for successful recovery will print the progress and output summary:

```text
Config loaded

Enriching event...

Enriched event 2dfeb9b7-5a87-4214-8a97-a8b23176856b

...

Total Lines: 156, Recovered: 142
```

By default no enrichments are configured, but local-only (ones that do not depend on upstream resource storage) enrichments can be enabled by setting `--enrichments` flag:

```text
snowplow-event-recovery run --config $PWD/scenarios.json -o /tmp -i $PWD/input/ --resolver $PWD/resolver.json --enrichments $PWD/enrichments



Total Lines: 3, Recovered: 0

OK! Successfully ran recovery.
```

---

# Troubleshoot the manual event recovery process
> Diagnose and resolve process failures, recovery failures, and unexpected behavior in event recovery jobs.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/troubleshooting/

In order to verify the process is running properly there are several locations that can be monitored, depending on your runtime environment these are: datasinks (for recovery job processed output): `failedOutput` (S3/GCS bucket), `unrecoverableOutput` (S3/GCS bucket), `output` (Kinesis/PubSub streams) and job runners (tracking job status and processing in real-time).

## Possible Failures

There are two main reasons why recovery process may fail. Process Failures preventing recovery process to proceed and Recovery Failures that mean recovery flow cannot be applied to given bad row/line. Another set of issues is unexpected behavior of the application. The three failure types need to be handled differently.

### Process Failures

Process failures are usually caused by misconfiguration. They usually happen as the job is submitted to its runner (Apache Beam, Spark, Flink). Reported failures may differ, but usually they boil down to:

#### Missing Resources

This means that resources set as input or output for recovery do not exist or are inaccessible to application.

**Action:** Check the existence of resources provided to job configuration and their access policies.

#### Configuration Issues

The largest group of issues causing early process termination are configuration issues. They are reported in CLI as the job is submitted.

**Action:** Make sure the configuration is well formed and it is conforming its schema.

### Recovery Failures

Definitely the largest and the most common issue with automated recovery process are recovery failures. These are runtime errors that prevent application from recovering given lines / bad rows. These failures will produce values under `failedOutput` directory.

Recovery error is yet another bad row type that is able to be recovered. For convenience recovery errors use original configurations and do not require another recovery flows configured especially for them.

Recovery error contain additional information pointing to:

- recovery flow name
- number of recoveries attempted
- original bad row

#### Missing recovery configuration

Happens when trying to recover a line for which no recovery flow configuration can be matched.

**Action:** Check whether supplied configuration contains a proper mapping for problematic bad row instance including its schema key and conditions that are applied before choosing configuration.

#### Replacement Failures

Actual replacement specified with Replace Step has failed. Reason will be supplied along with exported `RecoveryError` bad row under `unrecoveredOutput`.

**Action:** consult the exported message. This likely means that configuration supplied to modify the source bad row needs adjusting. Either with another recovery flow (when faulty row is a special case) or direct adjustment to recovery flow supplied with `RecoveryError`.

#### Cast Failures

Much like Replacement Failures, Cast Failures occur when modifying actual value with a Cast Step has failed. The reason is supplied along with exported `RecoveryError` bad row under `unrecoveredOutput`.

**Action:** consult the exported message. This likely means that configuration supplied to modify the source bad row needs adjusting. Either with another recovery flow (when faulty row is a special case) or direct adjustment to recovery flow supplied with `RecoveryError`.

#### Unrecoverable Bad Row Type

Some bad row types are marked as unrecoverable and therefore even with an existing configuration they will not be transformed. They are outputted to `unrecoverable`.

**Action:** If you think that specific bad row type should be recoverable consult Extending Recovery and consider submitting a Pull Request.

#### Invalid JSON Format

Happens whenever input data is not a proper JSON format.

**Action:** Check that input JSON file contains a proper JSON object in each line.

#### Invalid Data Format

Happens whenever input data is not a valid `BadRow` JSON format.

**Action:** Check that input JSON file contains a proper `BadRow` JSON object in each line.

#### Codec Failures

This set of failures should not really occur but means that there was an error serializing data with thrift (ThriftFailure) or Base64 coding. Some of these issues might be related to runtime locale.

**Action:** Consult supplied error message outputted along with BadRow into `unrecoveredOutput`. Verify that bad row producer source has the same locale as recovery worker – UTF-8.

### Unexpected Behavior

This is hopefully the rarest but the least specific set of issues that may occur. In case any unexpected behavior happens consider submitting a GitHub issue.

---

# Recover loaded events from a failed events table with SQL
> Learn how to recover failed events from your failed events table back into your good events table using SQL queries.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/recover-with-sql/

This guide will walk you through the process of recovering [failed events](/docs/fundamentals/failed-events/) from your failed events table back into your good events table. Failed events occur when [events](/docs/fundamentals/events/) don't pass validation during processing, typically due to [schema](/docs/fundamentals/schemas/) violations or enrichment failures.

## Recovery process

Recovering failed events involves identifying the failure type, reconstructing the data, and reinserting it into your good events table. The process varies depending on the specific failure that occurred.

The main steps are:

1. **Identify the type of failure**: there are many possible reasons an event can fail and end up in your failed events table. The failed events table only includes schema violations and enrichment failures, but there are still many different types:

   - Missing schema - the schema referenced (or specific version of that schema) isn't present in your prod Iglu repository.
   - Missing required property.
   - Extra properties that are not allowed in this schema version.
   - Referencing an incorrect schema version.
   - Incorrect data type.
   - Breaching a validation rule (a `min`, `max`, `minLength`, `maxLength`, `pattern`, `format`, `enum`, etc.)

2. **Reconstruct the offending row**: how you do this will depend on the type of failure that has occurred, and your specific business logic. For example, if you have a missing schema, because you started tracking before pushing the schema to Prod, then you should push the schema to Prod, and then essentially just select the values from the failure entity. However, if you have a string that is too long for the `maxLength` validation rule for example, then you must choose how to resolve this (trim to the max allowed value, or push a new schema with an increased limit, and change the schema version in the offending row from the original to the new version).

3. **Create a prep `SELECT` statement**: create a statement that addresses the issue. You can test this by attempting to `UNION` your prep table to a `SELECT` from the good `events` table to check for type inconsistencies.

4. **Create an `INSERT` command**: add the repaired row back to the main `events` table, leaving all other values unaltered. You can either type this out by hand, or if you don't want to do this (your events table may have hundreds of columns which is time consuming and error prone to type out manually) you can use the `INFORMATION_SCHEMA.COLUMNS` metadata table in your warehouse to construct the `INSERT` command programmatically.

## Identifying the nature of the failed event

In your failed events table, there is a column called `contexts_com_snowplowanalytics_snowplow_failure_1` which contains information about why the event failed. All the other columns are as-is when the event was tracked and enriched.

The following query will show you all the failures in your failed events table, and make the fields easier to read:

**BigQuery:**

```sql
SELECT

    app_id,

    event_name,

    event_version,

    contexts_com_snowplowanalytics_snowplow_failure_1,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array

    contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,

    SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,

        '/')[0], ':')[1] AS schema_vendor,

    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,

    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version

  FROM `snowplow_failed.events` -- Use your failed events schema

  WHERE 1=1

    AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day) -- include a date range for when the failed events occurred

    AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL
```

**Snowflake:**

```sql
SELECT

    app_id,

    event_name,

    event_version,

    contexts_com_snowplowanalytics_snowplow_failure_1,

    contexts_com_snowplowanalytics_snowplow_failure_1[0]:data AS data,  -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array

    contexts_com_snowplowanalytics_snowplow_failure_1[0]:errors AS errors,

    contexts_com_snowplowanalytics_snowplow_failure_1[0]:failure_type AS failure_type,

    contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema AS schema,

    contexts_com_snowplowanalytics_snowplow_failure_1[0]:timestamp AS timestamp,



    SPLIT_PART(SPLIT_PART(contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema::STRING, '/', 1), ':', 2) AS schema_vendor,

    SPLIT_PART(contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema::STRING, '/', 2) AS schema_name,

    SPLIT_PART(contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema::STRING, '/', 4) AS schema_version



FROM SNOWPLOW_FAILED.EVENTS -- Use your failed events schema

WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL '7 DAY' -- include a date range for when the failed events occurred

  AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL;
```

**Databricks:**

```sql
SELECT

  app_id,

  event_name,

  event_version,

  contexts_com_snowplowanalytics_snowplow_failure_1,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array

  contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,

  SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,

      '/')[0], ':')[1] AS schema_vendor,

  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,

  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version

FROM snowplow_failed.events -- Use your failed events schema

WHERE 1=1

  AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7) -- include a date range for when the failed events occurred

  AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL
```

***

You can add additional filters to this query to isolate the specific failures you're interested in recovering:

**BigQuery:**

```sql
WITH

  failed_events AS (

  SELECT

    app_id,

    event_name,

    event_version,

    contexts_com_snowplowanalytics_snowplow_failure_1,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,

    contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,

    SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,

        '/')[0], ':')[1] AS schema_vendor,

    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,

    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version

  FROM `snowplow_failed.events` -- Use your failed events schema

  WHERE 1=1

    AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day)

    AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL

  ORDER BY timestamp DESC)

  SELECT

    event_name,

    schema,

    schema_name,

    schema_version,

    schema_vendor,

    data,

    errors,

    data.user_name,

    data.user_id

  FROM failed_events

  WHERE 1=1

    AND schema_name = 'user_entity'

    AND app_id = 'test-app'

  ORDER BY timestamp DESC
```

**Snowflake:**

```sql
WITH failed_events AS (

  SELECT

    app_id,

    event_name,

    event_version,

    f.value AS failure_context,

    f.value:data AS data,

    f.value:errors AS errors,

    f.value:failure_type AS failure_type,

    f.value:schema AS schema,

    f.value:timestamp AS timestamp,

    SPLIT_PART(SPLIT_PART(f.value:schema::string, '/', 1), ':', 2) AS schema_vendor,

    SPLIT_PART(f.value:schema::string, '/', 2) AS schema_name,

    SPLIT_PART(f.value:schema::string, '/', 4) AS schema_version

  FROM SNOWPLOW_FAILED.EVENTS,  -- Use your failed events schema

       LATERAL FLATTEN(input => contexts_com_snowplowanalytics_snowplow_failure_1) f -- using flatten allows for instances where multiple entities have failed on the same event

  WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL '7 DAYS'

)

SELECT

  event_name,

  schema,

  schema_name,

  schema_version,

  schema_vendor,

  data,

  errors,

  data:user_name AS user_name,

  data:user_id AS user_id,

FROM failed_events

WHERE schema_name = 'user_entity'

  AND app_id = 'test-app'

ORDER BY timestamp DESC;
```

**Databricks:**

```sql
WITH

failed_events AS (

SELECT

  app_id,

  event_name,

  event_version,

  contexts_com_snowplowanalytics_snowplow_failure_1,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,

  contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,

  SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,

      '/')[0], ':')[1] AS schema_vendor,

  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,

  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version

FROM snowplow_failed.events -- Use your failed events schema

WHERE 1=1

  AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7)

  AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL

ORDER BY timestamp DESC)

SELECT

  event_name,

  schema,

  schema_name,

  schema_version,

  schema_vendor,

  data,

  errors,

  get_json_object(data, '$.user_name') as user_name,

  get_json_object(data, '$.user_id') as user_id

FROM failed_events

WHERE 1=1

  AND schema_name = 'user_entity'

  AND app_id = 'test-app'

ORDER BY timestamp DESC
```

***

Here we are filtering for our `test-app` app ID, and the specific schema failures we're looking for (`WHERE schema_name = 'user_entity'`).

The `data` field is a JSON object of the data that was included in the event, untouched from when it was tracked. We can access these using dot notation on the `data` object - e.g. `data.user_name`.

![Failed events query results showing ResolutionError failures](/assets/images/failed-events-results-table-7e2bc8645de5d51b4d45e42af3bfdd93.png)

In this example, we can see that in the `schema_version` field, the version is `2-0-1` when the most up to date version of this schema is `2-0-0` and we've therefore had a `ResolutionError` in the `failure_type` column - meaning that Iglu wasn't able to find the schema that was included in the event. Therefore, we need to reconstruct the entity columns, with the correct schema version and the correct corresponding column name for that version. This is a relatively straightforward fix, as we don't have to mutate the data in place at all.

## Reconstructing the failed event row

Even though this is a fairly straightforward fix (essentially in this case it is a passthrough) we must use some knowledge of how Snowplow constructs data in our good `events` table in order to correctly reconstruct the event before inserting.

Key principles for reconstructing events:

- The properties types defined in the JSON schema for an event or entity are deterministically converted into column types in the data warehouse.

- Custom events are analogous to JSON objects - key-value pairs of data on the event. The values in this object can be primitive types (strings, numbers, boolean values etc) or complex types (objects and arrays). The keys become the column names.

- Custom entities are stored as arrays of objects - this is because it is possible (and common) to attach multiple of the same entity to a single event - e.g. an array of `product` entities being attached to a `product_list` event. So even if there is only one of a given entity attached to an event, they are always stored as arrays in the data warehouse. Each entity object is semantically identical to that of a custom event i.e. a key-value object.

- The `contexts_com_snowplowanalytics_snowplow_failure_1` column is itself an array as it is an entity, and a single event can have multiple failures, each that should be addressed before reinserting.

- The target column names also follow a deterministic naming convention:

  - Custom event columns are of the format `unstruct_event_{vendor_name}_{schema_name}_{version_number | major_version_numer}`.
  - Custom entity columns are of the format `contexts_{vendor_name}_{schema_name}_{version_number | major_version_numer}`.
  - Column names also are "snake-ified", so camelCase names in your schemas will be converted to snake\_case, and dots (`.`) in your vendor name will also be snake-ified.

## Creating a prep query

In this example, the `user_entity` is obviously an entity, so will be in an array. Depending on the data warehouse you are using, the `data` object will be stored differently.

- BigQuery stores the `data` object as a JSON type.
- Snowflake uses a VARIANT type.
- Databricks uses a `STRING` type to represent the JSON object.

Regardless of the data warehouse in question, the repaired object should be an array containing a single struct value, and the individual values must be cast to their correct types from the value stored in the failed event object.

**BigQuery:**

```sql
WITH prep AS (

  SELECT

    contexts_com_snowplowanalytics_snowplow_failure_1

  FROM `snowplow_failed.events`

  WHERE 1=1

    AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day)

    AND app_id = 'test-app'

    AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = 'iglu:com.example/user_entity/jsonschema/2-0-1' -- filter for my offending schema

)

SELECT

  ARRAY( -- since this is an entity, it must be an array

    SELECT

      STRUCT(

        '2-0-0' AS _schema_version, -- set the schema version to the correct entity version for BigQuery Loader V2

        INT64(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_id) AS user_id, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array

        STRING(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_name) AS user_name -- use the `INT64(...)` and the `STRING(...)` functions to cast the raw JSON values to their appropriate types

      )

    )

  AS contexts_com_example_user_entity_2_0_0 -- correctly name the column name to be inserted

FROM prep;
```

**Snowflake:**

```sql
WITH prep AS (

    SELECT contexts_com_snowplowanalytics_snowplow_failure_1

    FROM SNOWPLOW_FAILED.EVENTS

    WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL '7 DAY'

    AND app_id = 'test-app'

    AND contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema = 'iglu:com.example/user_entity/jsonschema/2-0-1'

  )

SELECT

  ARRAY_CONSTRUCT( -- since this is an entity, it must be an array

   OBJECT_CONSTRUCT(

   '_schema_version', '2-0-0', -- set the schema version to the correct entity version for Snowflake Streaming Loader

   'user_id', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_id::INT, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array

   'user_name', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_name::STRING  -- use the `...::INT` and the `...::STRING` functions to cast the raw JSON values to their appropriate types

   )

   ) AS CONTEXTS_COM_EXAMPLE_USER_ENTITY_2

FROM prep
```

**Databricks:**

```sql
WITH prep AS (

SELECT

  contexts_com_snowplowanalytics_snowplow_failure_1

FROM snowplow_failed.events

WHERE 1=1

  AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7)

  AND app_id = 'test-app'

  AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = 'iglu:com.example/user_entity/jsonschema/2-0-1' -- filter for my offending schema

)

SELECT

ARRAY( -- since this is an entity, it must be an array

    NAMED_STRUCT(

      '_schema_version', '2-0-0', -- set the schema version to the correct entity version for Databricks

      'user_id', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, '$.user_id') as BIGINT), -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array

      'user_name', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, '$.user_name') as STRING) -- CAST as BIGINT and STRING to cast the raw JSON string values to their appropriate types

    )

  )

AS contexts_com_example_user_entity_2_0_0 -- correctly name the column name to be inserted

FROM prep;
```

***

## Reinserting the repaired rows

Now we have successfully identified our problematic rows and corrected them, we can now insert them back to the events table. While you can hand write an `INSERT` statement for the rows, the Snowplow events tables tend to have very large and complex table schemas, with hundreds or thousands of columns, with many complex nested columns. This process is time consuming and error prone, so we recommend using the `INFORMATION_SCHEMA` tables to programmatically construct the `INSERT` command before running it.

There are some complexities to bear in mind when doing this:

- There may well be columns that are not in the failed events table that are in the good events table. Your insert command must include `CAST(NULL AS ... )` in place for all of those columns.
- You must ensure that you have the correct structure for your repaired columns as otherwise they will not insert and the warehouse will reject the inserted rows.
- In newer versions of Snowplow data warehouse loaders (such as Snowflake Streaming Loader and the BigQuery Loader V2), entity columns also have a `_schema_version` column. This must also be set for reprocessed entity columns in order for the insert to be successful.
- If the recovered events are net-new to your good events table (say you've encountered failures on the first time using this schema) then that column won't exist in your good events table yet. We'd recommend sending a hand crafted good event for that schema to your prod pipeline **first**. This will allow the Snowplow loader to correctly create the column in your good events table so it is ready to be inserted into.

> **Info:** We would recommend using a staging table to insert the repaired rows before running this process on your main events table. This will allow you to review the repaired rows before inserting them into your main events table. To do so you can take a cut of your main events table, and run the `INSERT` command against that table, and verify you get the behavior you expect.

Below are scripts for BigQuery and Snowflake to create you the `INSERT` command:

**BigQuery:**

```sql
-- Schema-aware INSERT generator for Snowplow failed events reprocessing



WITH target_table_columns AS ( -- this CTE finds all the columns in the good events table

  SELECT

    ordinal_position,

    column_name,

    data_type,

    is_nullable

  FROM `snowplow_good.INFORMATION_SCHEMA.COLUMNS`

  WHERE table_name = 'events'

    AND table_schema = 'snowplow_good'

),



failed_table_columns AS ( -- this CTE finds all the columns in the failed events table

  SELECT

    column_name,

    data_type,

    is_nullable

  FROM `snowplow_failed.INFORMATION_SCHEMA.COLUMNS`

  WHERE table_name = 'events'

    AND table_schema = 'snowplow_failed'

),



schema_comparison AS ( -- this CTE joins the previous two, and labels them if they are missing

  SELECT

    t.ordinal_position,

    t.column_name,

    t.data_type AS target_type,

    f.data_type AS failed_type,

    t.is_nullable,

    CASE

      WHEN f.column_name IS NULL THEN 'MISSING_IN_FAILED'

      ELSE 'MATCH'

    END AS status

  FROM target_table_columns t

  LEFT JOIN failed_table_columns f ON t.column_name = f.column_name

),



column_mappings AS (   -- this CTE checks if columns are missing,

  SELECT               -- handles those missing columns by inserting NULLs

    ordinal_position,  -- and also inserts your repaired event columns

    column_name,

    target_type,

    status,

    CASE

      -- Handle your specific replaced entity column

      WHEN column_name = 'contexts_com_example_user_entity_2_0_0' THEN -- isolate the target column

      -- paste in your reprocess logic as a string (including newline characters (`\n`) if you wish)

        '''array(select struct(

        \'2-0-0\' AS _schema_version,

        int64(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_id) AS user_id,

        string(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_name) AS user_name

        )) AS contexts_com_example_user_entity_2_0_0'''



      -- Exclude the failure entity from source

      WHEN column_name = 'contexts_com_snowplowanalytics_snowplow_failure_1' THEN

        'cast(null as array<struct<schema string, data json>>) AS contexts_com_snowplowanalytics_snowplow_failure_1'



      -- Handle the load_tstamp column so that the inserted rows have the correct timestamp

      WHEN column_name = 'load_tstamp' THEN 'current_timestamp() AS load_tstamp'



      -- Handle missing columns with appropriate NULL casting

      WHEN status = 'MISSING_IN_FAILED' THEN

        CASE

          -- Entity columns

          WHEN STARTS_WITH(lower(column_name), 'contexts_') THEN

            FORMAT('cast(null as %s) AS %s', target_type, column_name)

          -- Custom event columns

          WHEN STARTS_WITH(lower(column_name), 'unstruct_') THEN

            FORMAT('cast(null as %s) AS %s', target_type, column_name)

          ELSE

            FORMAT('cast(null as %s) AS %s', target_type, column_name)

        END



      -- Handle type mismatches with explicit casting

      WHEN status = 'TYPE_MISMATCH' THEN

        FORMAT('cast(%s as %s) AS %s', column_name, target_type, column_name)



      -- Pass through matching columns

      ELSE column_name

    END AS select_expression

  FROM schema_comparison

),



select_clause AS ( -- this CTE lists all the columns to go in the `INSERT` statement

  SELECT STRING_AGG(select_expression, ',\n  ' ORDER BY ordinal_position) AS columns_sql

  FROM column_mappings

),



generated_insert AS (

  SELECT FORMAT("""INSERT INTO `snowplow_good.events`

WITH prep AS (

  SELECT

    * -- include all columns from the failed events table

  FROM `snowplow_failed.events`

  WHERE DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day)

    AND app_id = 'test-app'

    AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = 'iglu:com.example/user_entity/jsonschema/2-0-1'

)

SELECT

  %s

FROM prep;""",

    columns_sql

  ) AS insert_statement

  FROM select_clause

)

-- Output the complete INSERT statement

SELECT

  insert_statement AS generated_sql

FROM generated_insert
```

**Snowflake:**

```sql
-- Schema-aware INSERT generator for Snowplow failed events reprocessing



WITH target_table_columns AS ( -- this CTE finds all the columns in the good events table

  SELECT

    ordinal_position,

    column_name,

    data_type,

    is_nullable

  FROM SNOWPLOW_GOOD_DATABASE.INFORMATION_SCHEMA.COLUMNS -- Use the database which has your SNOWPLOW_GOOD schema

  WHERE lower(table_name) = 'events'

    AND lower(table_schema) = 'snowplow_good'

),



failed_table_columns AS ( -- this CTE finds all the columns in the failed events table

  SELECT

    column_name,

    data_type,

    is_nullable

  FROM SNOWPLOW_FAILED_DATABASE.INFORMATION_SCHEMA.COLUMNS   --Use the database which has your SNOWPLOW_FAILED schema

  WHERE lower(table_name) = 'events'

    AND lower(table_schema) = 'snowplow_failed'

),



schema_comparison AS ( -- this CTE joins the previous two, and labels them if they are missing

  SELECT

    t.ordinal_position,

    t.column_name,

    t.data_type AS target_type,

    f.data_type AS failed_type,

    t.is_nullable,

    CASE

      WHEN f.column_name IS NULL THEN 'MISSING_IN_FAILED'

      ELSE 'MATCH'

    END AS status

  FROM target_table_columns t

  LEFT JOIN failed_table_columns f ON t.column_name = f.column_name

),



column_mappings AS (   -- this CTE checks if columns are missing,

  SELECT               -- handles those missing columns by inserting NULLs

    ordinal_position,  -- and also inserts your repaired event columns

    column_name,

    target_type,

    status,

    CASE

      -- Handle your specific replaced entity column

      WHEN column_name = 'CONTEXTS_COM_EXAMPLE_USER_ENTITY_2' THEN  -- isolate the target column

      -- paste in your reprocess logic as a string (newline characters (`\n`) also work)

        'ARRAY_CONSTRUCT(

        OBJECT_CONSTRUCT(

        ''_schema_version'', ''2-0-0'', -- set the schema version to the correct entity version for Snowflake Streaming Loader

        ''user_id'', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_id::INT,

        ''user_name'', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_name::STRING

        )

        ) AS CONTEXTS_COM_EXAMPLE_USER_ENTITY_2'



      -- Exclude the failure entity from source

      WHEN column_name = 'contexts_com_snowplowanalytics_snowplow_failure_1' THEN

        'CAST(NULL AS ARRAY<OBJECT<schema STRING, data VARIANT>>) AS contexts_com_snowplowanalytics_snowplow_failure_1'



      -- Handle the load_tstamp column so that the inserted rows have the correct timestamp

      WHEN column_name = 'load_tstamp' THEN 'CURRENT_TIMESTAMP() AS load_tstamp'



      -- Handle missing columns with appropriate NULL casting

      WHEN status = 'MISSING_IN_FAILED' THEN

        CASE

          -- Entity columns

          WHEN STARTSWITH(lower(column_name), 'contexts_') THEN

            CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)

          -- Custom event columns

          WHEN STARTSWITH(lower(column_name), 'unstruct_') THEN

            CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)

          -- Standard columns

          ELSE

            CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)

        END



      -- Handle type mismatches with explicit casting

      WHEN status = 'TYPE_MISMATCH' THEN

        CONCAT('CAST(', column_name, ' AS ', target_type, ') AS ', column_name)



      -- Pass through matching columns

      ELSE column_name

    END AS select_expression

  FROM schema_comparison

),



select_clause AS (  -- this CTE lists all the columns to go in the `INSERT` statement

  SELECT LISTAGG(select_expression, ',\n  ') WITHIN GROUP (ORDER BY ordinal_position) AS columns_sql

  FROM column_mappings

),



generated_insert AS (

  SELECT CONCAT(

    'INSERT INTO SNOWPLOW_GOOD.EVENTS

    WITH prep AS (

    SELECT * -- include all columns from the failed events table

    FROM SNOWPLOW_FAILED.EVENTS

    WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL ''7 DAY''

    AND app_id = ''test-app''

    AND contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema = ''iglu:com.example/user_entity/jsonschema/2-0-1''

)

SELECT

  ',

    columns_sql,

    '

    FROM prep;'

  ) AS insert_statement

  FROM select_clause

)



-- Output the complete INSERT statement

SELECT

  insert_statement AS generated_sql

FROM generated_insert;
```

**Databricks:**

```sql
WITH target_table_columns AS ( -- Find all columns in the good events table

SELECT

  ordinal_position,

  column_name,

  full_data_type,

  is_nullable

FROM snowplow_good.information_schema.columns

WHERE table_name = 'events'

  AND table_schema = 'snowplow_good'

),



failed_table_columns AS ( -- Find all columns in the failed events table

SELECT

  column_name,

  full_data_type,

  is_nullable

FROM snowplow_failed.information_schema.columns

WHERE table_name = 'events'

  AND table_schema = 'snowplow_failed'

),



schema_comparison AS ( -- Join and label missing columns

SELECT

  t.ordinal_position,

  t.column_name,

  t.full_data_type AS target_type,

  f.full_data_type AS failed_type,

  t.is_nullable,

  CASE

    WHEN f.column_name IS NULL THEN 'MISSING_IN_FAILED'

    WHEN t.full_data_type != f.full_data_type THEN 'TYPE_MISMATCH'

    ELSE 'MATCH'

  END AS status

FROM target_table_columns t

LEFT JOIN failed_table_columns f ON t.column_name = f.column_name

),



column_mappings AS ( -- Handle missing columns and insert repaired event columns

SELECT

  ordinal_position,

  column_name,

  target_type,

  status,

  CASE

    -- Handle your specific replaced entity column

    WHEN column_name = 'contexts_com_example_user_entity_2_0_0' THEN

      -- Paste in your reprocess logic as a string

      'ARRAY(NAMED_STRUCT(

      ''_schema_version'', ''2-0-0'',

      ''user_id'', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, ''$.user_id'') AS BIGINT),

      ''user_name'', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, ''$.user_name'') AS STRING)

      )) AS contexts_com_example_user_entity_2_0_0'



    -- Exclude the failure entity from source

    WHEN column_name = 'contexts_com_snowplowanalytics_snowplow_failure_1' THEN

      'CAST(NULL AS ARRAY<STRUCT<schema:STRING, data:STRING>>) AS contexts_com_snowplowanalytics_snowplow_failure_1'



    -- Handle the load_tstamp column

    WHEN column_name = 'load_tstamp' THEN 

      'CURRENT_TIMESTAMP() AS load_tstamp'



    -- Handle missing columns with appropriate NULL casting

    WHEN status = 'MISSING_IN_FAILED' THEN

      CASE

        -- Entity columns (contexts)

        WHEN LOWER(column_name) LIKE 'contexts_%' THEN

          CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)

        -- Custom event columns (unstruct)

        WHEN LOWER(column_name) LIKE 'unstruct_%' THEN

          CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)

        -- Standard columns

        ELSE

          CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)

      END



    -- Handle type mismatches with explicit casting

    WHEN status = 'TYPE_MISMATCH' THEN

      CONCAT('CAST(', column_name, ' AS ', target_type, ') AS ', column_name)



    -- Pass through matching columns

    ELSE column_name

  END AS select_expression

FROM schema_comparison

),



select_clause AS ( -- List all columns for the INSERT statement

SELECT 

  CONCAT_WS(',\n  ', COLLECT_LIST(select_expression)) AS columns_sql

FROM (

  SELECT ordinal_position, select_expression

  FROM column_mappings

  ORDER BY ordinal_position

)

),



generated_insert AS (

SELECT 

  CONCAT(

    'INSERT INTO snowplow_good.events\n',

    'WITH prep AS (\n',

    '  SELECT\n',

    '    * -- include all columns from the failed events table\n',

    '  FROM snowplow_failed.events\n',

    '  WHERE DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7)\n',

    '    AND app_id = ''test-app''\n',

    '    AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = ''iglu:com.example/user_entity/jsonschema/2-0-1''\n',

    ')\n',

    'SELECT\n  ',

    columns_sql,

    '\nFROM prep;'

  ) AS insert_statement

FROM select_clause

)



-- Output the complete INSERT statement

SELECT

insert_statement AS generated_sql

FROM generated_insert;
```

***

This will print out the correctly formatted `INSERT` statement. You can now copy this and run it against your data warehouse, and it will reinsert the newly repaired failed events back into your good events table.

---

# Configure the Collector
> View and configure Snowplow collector settings via Console UI or API, including cookie domains, paths, and HTTP headers.
> Source: https://docs.snowplow.io/docs/pipeline/collector/

Once your [event collector](/docs/fundamentals/) is set up, along with [trackers](/docs/sources/) to submit events to them, you may want to verify your collector configuration. This can take two forms, depending on your needs.

Read more about the technical aspects of the collector [here](/docs/api-reference/stream-collector/).

## Viewing collector configuration in Snowplow Console

The easiest way to access collector configuration is to view it within the Snowplow Console. To do that, after you log in click on _Pipeline Configuration_ under the respective pipeline's navigation menu:

![Snowplow Console showing a pipeline with the Configuration tab selected (indicated by a red arrow), and the left sidebar showing configuration sections including COLLECTOR (Collector domains, Tracker request paths, Collector settings), VALIDATION, COMPONENTS, and ALERTS.](/assets/images/image-1-8ca1a1d304dc7770448eb2698869a7ee.png)

Selecting the pipeline configuration tab

You can then view your configuration, with the default values displayed for empty fields:

![Snowplow Console Collector configuration page showing the Tracker request paths section with a table of default paths: POST at /com.snowplowanalytics.snowplow/tp2, Redirect at /r/tp2, and Iglu at /com.snowplowanalytics.iglu/v1.](/assets/images/image-2-d20f02e67f447d2499aeba129631d9ab.png)

Example collector configuration

This view is consuming the respective API that you can also access, as discussed in the next section.

## Consuming the collector configuration API

As a Snowplow customer you already benefit from 24x7 monitoring of pipeline collector health. If you wish to add collector monitoring to your internal monitoring systems nevertheless, the maintainable way to do this is to retrieve collector endpoints and other configuration values via the available API, then invoke your health checks on them.

### Authorization

To start using the API you’ll need authorization credentials.

First, you’ll need to:

- [Create a new user account for your API authorization in console](https://console.snowplowanalytics.com/users)
- Request the `clientid` and `clientsecret` for the API service from the [admin section in console](https://console.snowplowanalytics.com/credentials).

Once you have these you can exchange credentials for a token.

Here is an example CURL to use to fetch the token:

```bash
curl --request POST \

  --url 'https://id.snowplowanalytics.com/oauth/token' \

  --header 'content-type: application/x-www-form-urlencoded' \

  --data grant_type=password \

  --data username=USER@DOMAIN.COM \

  --data password='PASSWORD' \

  --data audience=https://snowplowanalytics.com/api/ \

  --data client_id='YOUR_CLIENT_ID' \

  --data client_secret='YOUR_CLIENT_SECRET'
```

This token will be needed in any request to the API in the form of `Authorization: Bearer {{token}}`

### Getting started

You can have a look at and interact with the collector configuration endpoint in the normative [API specification](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml#/configuration/getOrganizationsOrganizationidResourcesV1PipelinesPipelineidConfigurationCollector).

### Authorizing in the API documentation

To be able to post sample requests in the documentation you need to click the `Authorize` button at the top of the document and authorize with your token. The value for the token field in each individual requests is overwritten by this authorization.

The endpoint for the API is:

`https://console.snowplowanalytics.com/api/msc/v1/organizations/{organizationId}/resources/v1/pipelines/{pipelineId}/configuration/collector`

and the only supported HTTP verb is `GET`.

You can find your organization ID [on the **Manage organization** page](https://console.snowplowanalytics.com/settings) in Console.

Similarly, the pipeline ID can be found in the location bar as the second UUID there:

![Snowplow Console browser address bar showing a URL where the pipeline ID UUID is highlighted as the second UUID in the path.](/assets/images/image-4-716927604c876d4c0652077b2cc6071c.png)

The Pipeline ID is the second UUID in the location bar.

### Invoking the API

Invoking this API will return an object of the following form:

```json
{

    "cookieDomains":

    {

        "fallback": "mydomain.com",

        "domains": []

    },

    "paths": {

        "post": {

            "paths": []

        },

        "webhook": {

            "paths": []

        },

        "redirect": {

            "paths": [],

            "enabled": true

        }

    },

    "cookieAttributes": {

        "secure": true,

        "sameSite": "None",

        "httpOnly": false

    },

    "blockUnencrypted": false

}
```

The `cookieDomains` object is always expected to be available and holds two properties:

- `domains` is a non-optional list of domains for which collector cookies are being set. The value may be an empty list.
- `fallback` represents the default fallback cookie domain. The property is optional and if unavailable it defaults to `none`.

The `paths` object is always expected to be available and holds the following three non-optional properties:

- `post` contains a non-optional list of paths that accept Tracker Protocol 2 compatible POST requests

- `redirect` contains

  - a non-optional flag weather redirect-based tracking is enabled, and
  - a non-optional list of paths that support this kind of tracking

- `webhook` contains a non-optional list of paths for tracking events sent via a `GET` or `POST` request containing an [Iglu](https://github.com/snowplow/iglu)-compatible event payload.

The `cookieAttributes` object is always expected to be available and contains three non-optional properties:

- `secure` is a boolean value indicating whether secure connections should be enforced;

- `httpOnly` is a boolean value set to true if the cookie must be inaccessible to non-HTTP requests; and

- `sameSite` is a string that can take one of the following values:

  - `lax`
  - `none`
  - `strict`

Finally, `blockUnencrypted` is an optional boolean property indicating whether un-encrypted traffic should be allowed or not. If not available, the default is `false` (i.e. "do not block").

## Configuring the collector for Snowplow Self-Hosted users

After you have installed the [Collector](/docs/api-reference/stream-collector/) (either following the [Quick Start guide](/docs/get-started/self-hosted/) or [manually](/docs/api-reference/stream-collector/setup/)), you can follow the [reference page](/docs/api-reference/stream-collector/configure/) to configure it.

## HTTP request headers

The Collector will collect any standard HTTP headers and the values of these headers can be extracted during Enrichment. The [HTTP header extractor enrichment](/docs/pipeline/enrichments/available-enrichments/http-header-extractor-enrichment/) can be configured for the headers you wish to extract.

Additionally, the following two headers can be sent on requests:

| Header       | Allowed Values     | Description                                                                                                |
| ------------ | ------------------ | ---------------------------------------------------------------------------------------------------------- |
| Content-Type | `application/json` | See [MDN Content-Type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type)             |
| SP-Anonymous | `*`                | Enables Server Side Anonymization, preventing the User IP Address and Network User ID from being collected |

### Cookie header

The Collector will collect any cookie information sent in the `Cookie` HTTP header. Cookies can be attached to events using the [Cookie extractor enrichment](/docs/pipeline/enrichments/available-enrichments/cookie-extractor-enrichment/)

---

# Enriched stream TSV file format
> Understand the Tab Separated Values format of enriched Snowplow data streams with field definitions and analytics SDK references.
> Source: https://docs.snowplow.io/docs/pipeline/enriched-tsv-format/

The Snowplow pipeline outputs the enriched stream in a Tab Separated Values (TSV) format. As TSV files do not contain header information, this page exists to help users of the enriched stream understand what each value represents.

Additionally, Snowplow has a number of Analytics SDKs available which help parse the TSV records into JSON:

- [Analytics SDK Scala](https://github.com/snowplow/snowplow-scala-analytics-sdk)
- [Analytics SDK Python](https://github.com/snowplow/snowplow-python-analytics-sdk)
- [Analytics SDK .NET](https://github.com/snowplow/snowplow-dotnet-analytics-sdk)
- [Analytics SDK Javascript](https://github.com/snowplow-incubator/snowplow-js-analytics-sdk/)
- [Analytics SDK Golang](https://github.com/snowplow/snowplow-golang-analytics-sdk)

For explanations of what each field represents, please see the [Canonical Event Model](/docs/fundamentals/canonical-event/).

## TSV header

Copy this if you need a header to work with the Snowplow TSV files.

```text
app_id	platform	etl_tstamp	collector_tstamp	dvce_created_tstamp	event	event_id	txn_id	name_tracker	v_tracker	v_collector	v_etl	user_id	user_ipaddress	user_fingerprint	domain_userid	domain_sessionidx	network_userid	geo_country	geo_region	geo_city	geo_zipcode	geo_latitude	geo_longitude	geo_region_name	ip_isp	ip_organization	ip_domain	ip_netspeed	page_url	page_title	page_referrer	page_urlscheme	page_urlhost	page_urlport	page_urlpath	page_urlquery	page_urlfragment	refr_urlscheme	refr_urlhost	refr_urlport	refr_urlpath	refr_urlquery	refr_urlfragment	refr_medium	refr_source	refr_term	mkt_medium	mkt_source	mkt_term	mkt_content	mkt_campaign	contexts	se_category	se_action	se_label	se_property	se_value	unstruct_event	tr_orderid	tr_affiliation	tr_total	tr_tax	tr_shipping	tr_city	tr_state	tr_country	ti_orderid	ti_sku	ti_name	ti_category	ti_price	ti_quantity	pp_xoffset_min	pp_xoffset_max	pp_yoffset_min	pp_yoffset_max	useragent	br_name	br_family	br_version	br_type	br_renderengine	br_lang	br_features_pdf	br_features_flash	br_features_java	br_features_director	br_features_quicktime	br_features_realplayer	br_features_windowsmedia	br_features_gears	br_features_silverlight	br_cookies	br_colordepth	br_viewwidth	br_viewheight	os_name	os_family	os_manufacturer	os_timezone	dvce_type	dvce_ismobile	dvce_screenwidth	dvce_screenheight	doc_charset	doc_width	doc_height	tr_currency	tr_total_base	tr_tax_base	tr_shipping_base	ti_currency	ti_price_base	base_currency	geo_timezone	mkt_clickid	mkt_network	etl_tags	dvce_sent_tstamp	refr_domain_userid	refr_device_tstamp	derived_contexts	domain_sessionid	derived_tstamp	event_vendor	event_name	event_format	event_version	event_fingerprint	true_tstamp
```

## TSV properties

The following table lists the properties of the TSV file.

| Property Index | Property Name              |
| -------------- | -------------------------- |
| 0              | app\_id                    |
| 1              | platform                   |
| 2              | etl\_tstamp                |
| 3              | collector\_tstamp          |
| 4              | dvce\_created\_tstamp      |
| 5              | event                      |
| 6              | event\_id                  |
| 7              | txn\_id                    |
| 8              | name\_tracker              |
| 9              | v\_tracker                 |
| 10             | v\_collector               |
| 11             | v\_etl                     |
| 12             | user\_id                   |
| 13             | user\_ipaddress            |
| 14             | user\_fingerprint          |
| 15             | domain\_userid             |
| 16             | domain\_sessionidx         |
| 17             | network\_userid            |
| 18             | geo\_country               |
| 19             | geo\_region                |
| 20             | geo\_city                  |
| 21             | geo\_zipcode               |
| 22             | geo\_latitude              |
| 23             | geo\_longitude             |
| 24             | geo\_region\_name          |
| 25             | ip\_isp                    |
| 26             | ip\_organization           |
| 27             | ip\_domain                 |
| 28             | ip\_netspeed               |
| 29             | page\_url                  |
| 30             | page\_title                |
| 31             | page\_referrer             |
| 32             | page\_urlscheme            |
| 33             | page\_urlhost              |
| 34             | page\_urlport              |
| 35             | page\_urlpath              |
| 36             | page\_urlquery             |
| 37             | page\_urlfragment          |
| 38             | refr\_urlscheme            |
| 39             | refr\_urlhost              |
| 40             | refr\_urlport              |
| 41             | refr\_urlpath              |
| 42             | refr\_urlquery             |
| 43             | refr\_urlfragment          |
| 44             | refr\_medium               |
| 45             | refr\_source               |
| 46             | refr\_term                 |
| 47             | mkt\_medium                |
| 48             | mkt\_source                |
| 49             | mkt\_term                  |
| 50             | mkt\_content               |
| 51             | mkt\_campaign              |
| 52             | contexts                   |
| 53             | se\_category               |
| 54             | se\_action                 |
| 55             | se\_label                  |
| 56             | se\_property               |
| 57             | se\_value                  |
| 58             | unstruct\_event            |
| 59             | tr\_orderid                |
| 60             | tr\_affiliation            |
| 61             | tr\_total                  |
| 62             | tr\_tax                    |
| 63             | tr\_shipping               |
| 64             | tr\_city                   |
| 65             | tr\_state                  |
| 66             | tr\_country                |
| 67             | ti\_orderid                |
| 68             | ti\_sku                    |
| 69             | ti\_name                   |
| 70             | ti\_category               |
| 71             | ti\_price                  |
| 72             | ti\_quantity               |
| 73             | pp\_xoffset\_min           |
| 74             | pp\_xoffset\_max           |
| 75             | pp\_yoffset\_min           |
| 76             | pp\_yoffset\_max           |
| 77             | useragent                  |
| 78             | br\_name                   |
| 79             | br\_family                 |
| 80             | br\_version                |
| 81             | br\_type                   |
| 82             | br\_renderengine           |
| 83             | br\_lang                   |
| 84             | br\_features\_pdf          |
| 85             | br\_features\_flash        |
| 86             | br\_features\_java         |
| 87             | br\_features\_director     |
| 88             | br\_features\_quicktime    |
| 89             | br\_features\_realplayer   |
| 90             | br\_features\_windowsmedia |
| 91             | br\_features\_gears        |
| 92             | br\_features\_silverlight  |
| 93             | br\_cookies                |
| 94             | br\_colordepth             |
| 95             | br\_viewwidth              |
| 96             | br\_viewheight             |
| 97             | os\_name                   |
| 98             | os\_family                 |
| 99             | os\_manufacturer           |
| 100            | os\_timezone               |
| 101            | dvce\_type                 |
| 102            | dvce\_ismobile             |
| 103            | dvce\_screenwidth          |
| 104            | dvce\_screenheight         |
| 105            | doc\_charset               |
| 106            | doc\_width                 |
| 107            | doc\_height                |
| 108            | tr\_currency               |
| 109            | tr\_total\_base            |
| 110            | tr\_tax\_base              |
| 111            | tr\_shipping\_base         |
| 112            | ti\_currency               |
| 113            | ti\_price\_base            |
| 114            | base\_currency             |
| 115            | geo\_timezone              |
| 116            | mkt\_clickid               |
| 117            | mkt\_network               |
| 118            | etl\_tags                  |
| 119            | dvce\_sent\_tstamp         |
| 120            | refr\_domain\_userid       |
| 121            | refr\_device\_tstamp       |
| 122            | derived\_contexts          |
| 123            | domain\_sessionid          |
| 124            | derived\_tstamp            |
| 125            | event\_vendor              |
| 126            | event\_name                |
| 127            | event\_format              |
| 128            | event\_version             |
| 129            | event\_fingerprint         |
| 130            | true\_tstamp               |

---

# ASN lookup enrichment
> Flag bot traffic by checking autonomous system numbers against known bad ASN lists.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/

> **Note:** This enrichment is available since version 6.9.0 of Enrich.

This enrichment checks the autonomous system number (ASN) attached to an event against a configurable list of ASNs associated with bots, cloud providers, or abusive networks. When a match is found, the enrichment sets `likelyBot` to `true` on the ASN [entity](/docs/fundamentals/entities/) added by the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/).

This is useful for automatically flagging non-human traffic. Many bots and scrapers originate from well-known cloud hosting or data center ASNs, and community-maintained lists such as [cpuchain/bad-asn-list](https://github.com/cpuchain/bad-asn-list) track these.

> **Warning:** Many VPN services also use cloud hosting. As such, this enrichment might incorrectly flag VPN users as bots (hence the `likelyBot` and not `bot` designation).
>
> Depending on the nature of your business, VPN users might represent a meaningful portion of your traffic. If you find that to be the case, you should not rely on this enrichment as a _sole_ indicator of bots. Instead, you could use it to reinforce other indicators, e.g., unusually high number of page views in a short timeframe.

## Prerequisites

To use this enrichment, you need to enable the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) and configure it to produce ASN data.

If you are using the paid MaxMind database (`isp` field present in your IP lookup configuration), you don't need to do anything else.

If you are using the free MaxMind database, e.g., the one provided by Snowplow for CDI customers, your configuration will look like this:

```json
"geo": {

  "database": "GeoLite2-City.mmdb",

  "uri": "<database URI>"

}
```

Add the `asn` field:

```json
"geo": {

  "database": "GeoLite2-City.mmdb",

  "uri": "<database URI>"

},

"asn": {

  "database": "GeoLite2-ASN.mmdb",

  "uri": "<same URI value as for geo>"

}
```

## Configuration

The enrichment takes these parameters:

| Parameter         | Required | Description                                                                                     |
| ----------------- | -------- | ----------------------------------------------------------------------------------------------- |
| `botAsnsFile`     | ❌        | Location of a CSV file listing ASNs to flag as likely bots. Already provided for CDI customers. |
| `botAsns`         | ❌        | Inline list of ASNs to flag, merged with any entries from `botAsnsFile`.                        |
| `bypassPlatforms` | ❌        | Event platforms for which the enrichment should not run (e.g. server-side, IoT).                |

**Console:**

Configure the parameters in the Console enrichment editor. Keep the Console defaults for the `database` and `uri` fields. For example:

```json
{

  "botAsnsFile": {

    "uri": "<use default value from Console>",

    "database": "<use default value from Console>"

  },

  "botAsns": [

    { "asn": 123, "name": "ASN 123" },

    { "asn": 456 }

  ],

  "bypassPlatforms": ["srv"]

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/asn_lookups/jsonschema/1-0-0",

  "data": {

    "name": "asn_lookups",

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "enabled": true,

    "parameters": {

      "botAsnsFile": {

        "uri": "<your file location>",

        "database": "bad-asn-list.csv"

      },

      "botAsns": [

        { "asn": 123, "name": "ASN 123" },

        { "asn": 456 }

      ],

      "bypassPlatforms": ["srv"]

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### `botAsnsFile`

> **Tip:** If you're using Snowplow CDI, you don't need to configure this. Use the default values provided in Console.

Points to a CSV file with ASN numbers. The file should have a header row and use the format `number,name` (e.g., `174,"COGENT-174 - Cogent Communications, US"`). Only the number column is used for matching; the name column is for human readability and can be empty.

| Field      | Type   | Description                                                                                                        |
| ---------- | ------ | ------------------------------------------------------------------------------------------------------------------ |
| `uri`      | string | Base URI where the file is hosted. Supports `http:`, `s3:`, and `gs:` schemes. Must not end with a trailing slash. |
| `database` | string | The CSV filename.                                                                                                  |

You can use a community-maintained list such as [cpuchain/bad-asn-list](https://github.com/cpuchain/bad-asn-list). Host the CSV file in your own cloud storage to avoid depending on an external service at pipeline runtime.

### `botAsns`

An inline array of ASN objects. These are combined with any entries from `botAsnsFile`.

| Field  | Type    | Required | Description                                                              |
| ------ | ------- | -------- | ------------------------------------------------------------------------ |
| `asn`  | integer | ✅        | The autonomous system number.                                            |
| `name` | string  | ❌        | A human-readable label. Used only for clarity in the configuration file. |

### `bypassPlatforms`

An array of values that correspond to the `platform` field on events. Events with a matching platform skip this enrichment entirely, because it is expected for those platforms to originate from cloud or data center ASNs.

For example, server-side tracking (`"srv"`) and IoT (`"iot"`) events typically come from cloud providers, so flagging them as bots would produce false positives.

## Output

The [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) adds an ASN entity to events where ASN information is available. This enrichment modifies that entity by setting `likelyBot` to `true` when the ASN matches one from the configured list of bad ASNs.

This enrichment won't produce any output if:

- The IP lookup enrichment is not enabled
- ASN data is not enabled in the IP lookup enrichment configuration
- An event does not contain an IP address
- There is no ASN information for that IP address

### `asn`

**Type:** Entity

Schema for ASN entity generated by IP lookup enrichment

**Properties:**

| Property                | Description                                                                                             |
| ----------------------- | ------------------------------------------------------------------------------------------------------- |
| `number` _integer_      | _Required._ The autonomous system number associated with the IP address                                 |
| `organization` _string_ | _Optional._ The organization associated with the registered autonomous system number for the IP address |
| `likelyBot` _boolean_   | _Optional._ Set to true if the ASN belongs to hosting providers, data centers, etc.                     |

The `likelyBot` field isn't included in the entity if the event's platform is in `bypassPlatforms`.

---

# Bot detection enrichment
> Consolidate bot indicators from multiple enrichments into a single entity for easier filtering and analysis.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/

> **Note:** This enrichment is available since version 6.9.0 of Enrich.

Multiple enrichments can independently detect bots: [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/), [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/), and the [ASN lookup](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/). In addition, the [bot detection tracker plugin](/docs/sources/web-trackers/tracking-events/bot-detection/) for web can detect automated browsers client-side.

The bot detection enrichment consolidates these indicators into a single [entity](/docs/fundamentals/entities/). This lets you filter bot traffic in your data models, or drop bot events entirely using a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/#discard-events).

## How bot indicators are combined

The enrichment uses "any positive = bot" logic. If any of the enabled sources flags the event as coming from a bot, the event is classified as a bot. A negative result from one source doesn't override a positive result from another. This is because none of the existing enrichments can produce a strong "not a bot" result.

Each source contributes an indicator as follows:

| Source                | Flagged as bot when                                                                                                                                                                                             |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| YAUAA                 | `deviceClass` is `"Robot"`, `"Robot Mobile"`, or `"Robot Imitator"`, or `agentClass` is `"Robot"` or `"Robot Mobile"` in the [YAUAA entity](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) |
| IAB                   | `spiderOrRobot` is `true` in the [IAB entity](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/)                                                                                                 |
| ASN lookup            | `likelyBot` is `true` in the [ASN entity](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/)                                                                                              |
| Client-side detection | `bot` is `true` in the [client-side bot detection entity](/docs/sources/web-trackers/tracking-events/bot-detection/)                                                                                            |

For example, if YAUAA detects a bot based on user agent but IAB doesn't, the event is still classified as a bot. Similarly, if the ASN lookup flags the event based on a known bad ASN, that result stands regardless of what YAUAA or IAB report.

> **Note:** It's safe to enable all sources - `useYauaa`, `useIab`, `useAsnLookups`, `useClientSideDetection` - even if some of the underlying enrichments or plugins aren't in use. If a contributing source isn't enabled or its entity isn't present, that source is silently skipped.

## Configuration

The enrichment takes these parameters:

| Parameter                | Required | Description                                                                  |
| ------------------------ | -------- | ---------------------------------------------------------------------------- |
| `useYauaa`               | ✅        | Whether to use YAUAA `deviceClass`/`agentClass` as a bot signal.             |
| `useIab`                 | ✅        | Whether to use IAB spiders and robots as a bot signal.                       |
| `useAsnLookups`          | ✅        | Whether to use ASN lookups as a bot signal.                                  |
| `useClientSideDetection` | ✅        | Whether to use the client-side bot detection tracker plugin as a bot signal. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "useYauaa": true,

  "useIab": true,

  "useAsnLookups": true,

  "useClientSideDetection": true

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/bot_detection_enrichment_config/jsonschema/1-0-1",

  "data": {

    "name": "bot_detection_enrichment_config",

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "enabled": true,

    "parameters": {

      "useYauaa": true,

      "useIab": true,

      "useAsnLookups": true,

      "useClientSideDetection": true

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

The `useClientSideDetection` parameter is available since Enrich 6.10.0.

## Output

The enrichment produces a `bot_detection` entity that summarizes all bot indicators for the event. It adds this entity to all events, even when no bot is detected.

### `bot_detection`

**Type:** Entity

Schema for bot detection context generated by the Bot Detection enrichment

**Properties:**

| Property             | Description                                                      |
| -------------------- | ---------------------------------------------------------------- |
| `bot` _boolean_      | _Required._ Whether the event is classified as coming from a bot |
| `indicators` _array_ | _Required._ List of sources that indicated bot activity          |

---

# Campaign attribution enrichment
> Link events to marketing campaigns by extracting attribution data from query string parameters.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/

This enrichment can be used to link events to marketing campaigns, using the query string parameters.

When using online marketing campaigns to drive traffic to a website, it's often possible to find information in the query string parameters to identify the particular campaign, medium, and more.

A link for an online advertisement that brings users back to the site might look like:

```markup
https://www.acme.com/spring_offer_product?utm_source=influencer&utm_medium=blog&utm_campaign=spring_offer
```

The enrichment can extract values from the query string parameters, resulting in the following fields being added to the enriched event:

| field          | value          |
| -------------- | -------------- |
| `mkt_source`   | `influencer`   |
| `mkt_medium`   | `blog`         |
| `mkt_campaign` | `spring_offer` |

The configuration of the enrichment defines which parameters in the URL (e.g. `utm_source`) map to which fields in the event (e.g. `mkt_source`).

## Configuration

The enrichment takes one parameter, `fields`, which maps event marketing fields to URL query string parameter names. It must contain these keys:

| Key           | Required | Populates                       | Description                                                                                         |
| ------------- | -------- | ------------------------------- | --------------------------------------------------------------------------------------------------- |
| `mktMedium`   | ✅        | `mkt_medium`                    | Array of URL query string parameter names. Use `[]` if you don't need it.                           |
| `mktSource`   | ✅        | `mkt_source`                    | As above.                                                                                           |
| `mktTerm`     | ✅        | `mkt_term`                      | As above.                                                                                           |
| `mktContent`  | ✅        | `mkt_content`                   | As above.                                                                                           |
| `mktCampaign` | ✅        | `mkt_campaign`                  | As above.                                                                                           |
| `mktClickId`  | ❌        | `mkt_clickid` and `mkt_network` | Object mapping click parameter names to network names. See [below](#click-and-network-attribution). |

All five required keys must be present, even if you only want to extract some of them. Provide an empty array for any you don't need.

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "fields": {

    "mktMedium": ["utm_medium"],

    "mktSource": ["utm_source"],

    "mktTerm": ["utm_term"],

    "mktContent": ["utm_content"],

    "mktCampaign": ["utm_campaign"],

    "mktClickId": {

      "gbraid": "Google"

    }

  }

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/campaign_attribution/jsonschema/1-0-1",

  "data": {

    "name": "campaign_attribution",

    "vendor": "com.snowplowanalytics.snowplow",

    "enabled": true,

    "parameters": {

      "fields": {

        "mktMedium": ["utm_medium"],

        "mktSource": ["utm_source"],

        "mktTerm": ["utm_term"],

        "mktContent": ["utm_content"],

        "mktCampaign": ["utm_campaign"],

        "mktClickId": {

          "gbraid": "Google"

        }

      }

    }

  }

}
```

***

Here is what the above example does:

- take the value of `utm_medium=...` in the URL and place it into the `mkt_medium` field
- same for `utm_source` and `mkt_source`, and so on
- if `gbraid=...` is in the URL, take its value and place it into the `mkt_clickid` field, also setting `mkt_network` to `Google`

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### Supporting multiple parameters

What if some of your links use `utm_campaign=...` and some use `legacy_campaign=...`? You can configure more than one parameter name in the array, like so:

```json
    "parameters":{

      "fields":{

        "mktCampaign":[

          "utm_campaign",

          "legacy_campaign"

        ]

      }

    }
```

The same applies to the other configuration options `mktMedium`, `mktSource`, `mktTerm` and `mktContent`.

If the query string includes multiple acceptable parameters (e.g. both `utm_campaign` and `legacy_campaign`), the **first one listed in the configuration** will be used, not the first one present in the query string.

Results:

| Query string                           | Value of `mkt_campaign` |
| -------------------------------------- | ----------------------- |
| `utm_campaign=new`                     | `new`                   |
| `legacy_campaign=old`                  | `old`                   |
| `utm_campaign=new&legacy_campaign=old` | `new`                   |
| `legacy_campaign=old&utm_campaign=new` | `new`                   |

### Click and network attribution

You can define which URL parameters are used to populate the `mkt_clickid` field. For each parameter, the same configuration setting also defines what network — the `mkt_network` field — it corresponds to.

The default mappings are `gclid` for `Google`, `msclkid` for `Microsoft`, and the legacy `dclid` for `DoubleClick`.

> **Tip:** You can configure any parameter names or network names, including your custom ones.

This example shows how to define your own click attribution settings, using `mktClickId`:

- Add support for `wbraid` and `gbraid` [parameters](https://support.google.com/analytics/answer/11367152?hl=en), which will be mapped to `Google` as the corresponding `mkt_network`
- Override the `msclkid` parameter, so that it maps to `NotMicrosoft` as the marketing network instead of the default `Microsoft`
- Add a custom `xyzid` parameter that maps to the `XYZ` network
- The default mappings for `gclid` and `dclid` remain unaffected

```json
    "parameters": {

      "fields": {

        ...

        "mktClickId": {

          "wbraid": "Google",

          "gbraid": "Google",

          "msclkid": "NotMicrosoft",

          "xyzid": "XYZ"

        }

        ...

      }

    }
```

Use only one click id parameter per URL. If you add multiple to the same URL, e.g. both `wbraid` and `gbraid`, one of them will be picked arbitrarily.

Results:

| Query string            | Value of `mkt_clickid` | Value of `mkt_network` |
| ----------------------- | ---------------------- | ---------------------- |
| `wbraid=abc`            | `abc`                  | `Google`               |
| `msclkid=abc`           | `abc`                  | `NotMicrosoft`         |
| `xyzid=abc`             | `abc`                  | `XYZ`                  |
| `wbraid=abc&gbraid=def` | `abc` or `def` ⚠️      | `Google`               |

## Output

This enrichment populates the [marketing atomic event fields](/docs/fundamentals/canonical-event/#marketing-fields): `mkt_medium`, `mkt_source`, `mkt_term`, `mkt_content`, `mkt_campaign`, `mkt_clickid`, and `mkt_network`.

If you don't enable the enrichment, those fields will be empty.

---

# Cookie extractor enrichment
> Extract name-value pairs from first-party cookies and attach them to events as derived contexts.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/cookie-extractor-enrichment/

This enrichment extracts name-value pairs from cookies set on the Collector domain, attaching them to the event as derived entities. It uses the `Cookie` HTTP header.

When using a custom Collector domain, it's possible to capture values in first-party cookies set by other services such as ad servers or content management software (CMS). By capturing these cookie fields and attaching them to the event, you can use the data to better identify users of your website.

This table shows how the cookie and Collector domains interact, indicating whether or not the cookies can be accessed with this enrichment.

| Collector domain    | Cookie domain | Cookies extracted |
| ------------------- | ------------- | ----------------- |
| `c.snowplow.io`     | `acme.com`    | ❌                 |
| `t.acme.com`        | `c.acme.com`  | ❌                 |
| `t.acme.com`        | `acme.com`    | ✅                 |
| `sp.track.acme.com` | `acme.com`    | ✅                 |

## Configuration

The enrichment takes one parameter:

| Parameter | Required | Description                                         |
| --------- | -------- | --------------------------------------------------- |
| `cookies` | ✅        | List of cookie keys for which to extract the value. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "cookies": ["sp"]

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/cookie_extractor_config/jsonschema/1-0-0",

  "data": {

    "name": "cookie_extractor_config",

    "vendor": "com.snowplowanalytics.snowplow",

    "enabled": true,

    "parameters": {

      "cookies": ["sp"]

    }

  }

}
```

***

The example configuration captures the Collector's own `sp` cookie value. In practice, you would probably want to extract other more valuable cookies available on the company domain.

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).
>
> To test first-party cookies, you'll need to run Micro locally and [configure a custom DNS resolution rule](/docs/testing/snowplow-micro/local/remote-usage/#locally-resolving-an-existing-domain-name-to-micro).

## Output

For **each** key listed in the configuration, an `http_cookie` entity is added to the enriched event. Each entity contains only one key/value pair.

### `http_cookie`

**Type:** Entity

Schema for a single HTTP cookie, as defined in RFC 6265

**Properties:**

| Property         | Description |
| ---------------- | ----------- |
| `name` _string_  | _Required._ |
| `value` _string_ | _Required._ |

---

# Cross-navigation enrichment
> Parse cross-navigation data from query string parameters to track user journeys across domains and platforms.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/

This enrichment parses the extended cross-navigation format in the `_sp` query string parameter, and attaches a `cross_navigation` entity to an event.

Check out the [cross-navigation](/docs/events/cross-navigation/) page to learn why this can be useful.

The extended cross-navigation format is `_sp={domainUserId}.{timestamp}.{sessionId}.{subjectUserId}.{sourceId}.{sourcePlatform}.{reason}`. The `domainUserId` and `timestamp` fields will always be present, but some of the other fields may be null or empty. They're configured within the tracker.

If this enrichment isn't enabled, Enrich parses the `_sp` query string parameter according to the short format, `_sp={domainUserId}.{timestamp}`

> **Note:** The pipeline will always populate the `refr_domain_userid` and `refr_dvce_tstamp` enriched event fields from the cross-navigation query string, even if this enrichment isn't enabled.

## Configuration

This enrichment has no configurable parameters. Enable it in Console.

For Self-Hosted, [provide the following JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/):

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/cross_navigation_config/jsonschema/1-0-0",

  "data": {

    "enabled": false,

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "name": "cross_navigation_config"

  }

}
```

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Input

This enrichment extracts the `_sp` query string parameter from the `page_url` field from the Snowplow event.

## Output

Enrich always populates the `refr_domain_userid` and `refr_dvce_tstamp` event fields when possible, regardless of whether the cross-navigation enrichment is enabled.

The enrichment adds a `cross_navigation` entity to the enriched event.

### `cross_navigation`

**Type:** Entity

Schema for a user and session identifiers entity included in cross platform or cross domain navigation.

**Properties:**

| Property                  | Description                                                                                                                                               |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `user_id` _null_          | _Optional._ The business user ID, the user was identified in the source app.                                                                              |
| `domain_user_id` _string_ | _Required._ Domain user ID of the source app from which a link was followed. The corresponding atomic property is named refr\_domain\_userid.             |
| `session_id` _null_       | _Optional._ Session ID of the source app from which a link was followed.                                                                                  |
| `timestamp` _string_      | _Required._ Timestamp of the link click in the source app which triggered this navigation. The corresponding atomic property is named refr\_dvce\_tstamp. |
| `source_id` _null_        | _Optional._ ID of the source app where the link that started this navigation originated from.                                                             |
| `source_platform` _null_  | _Optional._ Platform of the source app (e.g., mob, web, tv). Must be one of: `web`, `mob`, `pc`, `srv`, `app`, `tv`, `cnsl`, `iot`, `headset`,``          |
| `reason` _null_           | _Optional._ Cross navigation linking information/identifier.                                                                                              |

---

# Currency conversion enrichment
> Convert transaction values to a base currency using Open Exchange Rates API for standardized reporting.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/

> **Tip:** We recommend managing currency conversion downstream instead of using this enrichment. For example, you could bring currency exchange rate information into your data warehouse and join that data with your Snowplow data.

This legacy enrichment uses [Open Exchange Rates](https://openexchangerates.org/) to convert the currencies used in transactions. It requires an Open Exchange Rates account and API key.

This enrichment only works with the legacy `tr_` and `ti_` [ecommerce atomic event fields](/docs/fundamentals/canonical-event/#legacy-ecommerce-fields). Also, it can only use the exchange rate from the end of the day prior to the event's `collector_tstamp`. For these reasons, we recommend handling currency conversion downstream instead of using this enrichment.

## Configuration

The enrichment takes these parameters:

| Parameter      | Required | Description                                                                                                                                                                                     |
| -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `apiKey`       | ✅        | Open Exchange Rates API key.                                                                                                                                                                    |
| `baseCurrency` | ✅        | Currency to convert all transaction values to.                                                                                                                                                  |
| `rateAt`       | ✅        | Determines which exchange rate will be used. Only `EOD_PRIOR` is supported, meaning that the enrichment uses the exchange rate from the end of the day prior to the event's `collector_tstamp`. |
| `accountType`  | ✅        | Level of Open Exchange Rates account. Must be `DEVELOPER`, `ENTERPRISE`, or `UNLIMITED`.                                                                                                        |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "accountType": "DEVELOPER",

  "apiKey": "{{KEY}}",

  "baseCurrency": "USD",

  "rateAt": "EOD_PRIOR"

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/currency_conversion_config/jsonschema/1-0-0",

  "data": {

    "enabled": false,

    "vendor": "com.snowplowanalytics.snowplow",

    "name": "currency_conversion_config",

    "parameters": {

      "accountType": "DEVELOPER",

      "apiKey": "{{KEY}}",

      "baseCurrency": "USD",

      "rateAt": "EOD_PRIOR"

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Input

This enrichment uses the following fields :

- `tr_currency`
- `tr_total`
- `tr_tax`
- `tr_shipping`
- `ti_currency`
- `ti_price`

## Output

This enrichment updates the following fields of the atomic event:

| Field              | Purpose                                                                                          |
| ------------------ | ------------------------------------------------------------------------------------------------ |
| `base_currency`    | Base currency code according to [ISO\_4217](https://en.wikipedia.org/wiki/ISO_4217#Active_codes) |
| `tr_total_base`    | Total amount of transaction in base currency                                                     |
| `tr_tax_base`      | Tax applied in base currency                                                                     |
| `tr_shipping_base` | Shipping cost in base currency                                                                   |
| `ti_price_base`    | Item price in base currency                                                                      |

---

# Custom API request enrichment
> Enrich events with data from external HTTP APIs by making custom API requests during event processing.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/

The API request enrichment lets you add data to a Snowplow event via your own or third-party proprietary HTTP(S) API. Only basic access authentication is supported.

For example, use a common key like a user ID or an email address to add relevant information about a user to each event, before it gets written to your data store. The retrieved data is added as entities.

As with all enrichments, only one instance of it can be configured within your pipeline. This means you can only call one API during event processing.

## Configuration

For historical reasons, the configuration uses terms that are no longer used elsewhere in Snowplow.

The enrichment takes these parameters:

| Parameter       | Required | Description                                                                                       |
| --------------- | -------- | ------------------------------------------------------------------------------------------------- |
| `inputs`        | ✅        | Specifies the data points from the Snowplow event to use as keys when performing your API lookup. |
| `api`           | ✅        | Defines how the enrichment can access your API.                                                   |
| `outputs`       | ✅        | Specify how to process the returned JSON.                                                         |
| `cache`         | ✅        | Improves the enrichment's performance by storing values retrieved from the API.                   |
| `ignoreOnError` | ❌        | Whether to make the event fail if the API request fails.                                          |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "inputs": [

    {

      "key": "user",

      "json": {

        "field": "contexts",

        "schemaCriterion": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",

        "jsonPath": "$.userId"

      }

    },

    {

      "key": "client",

      "pojo": {

        "field": "app_id"

      }

    }

  ],

  "api": {

    "http": {

      "method": "GET",

      "uri": "http://api.acme.com/users/{{client}}/{{user}}?format=json",

      "timeout": 2000,

      "authentication": {

        "httpBasic": {

          "username": "xxx",

          "password": "yyy"

        }

      }

    }

  },

  "outputs": [

    {

      "schema": "iglu:com.acme/user/jsonschema/1-0-0",

      "json": {

        "jsonPath": "$.record"

      }

    }

  ],

  "cache": {

    "size": 3000,

    "ttl": 60

  },

  "ignoreOnError": false

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-2",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "name": "api_request_enrichment_config",

    "enabled": false,

    "parameters": {

      "inputs": [

        {

          "key": "user",

          "json": {

            "field": "contexts",

            "schemaCriterion": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",

            "jsonPath": "$.userId"

          }

        },

        {

          "key": "client",

          "pojo": {

            "field": "app_id"

          }

        }

      ],

      "api": {

        "http": {

          "method": "GET",

          "uri": "http://api.acme.com/users/{{client}}/{{user}}?format=json",

          "timeout": 2000,

          "authentication": {

            "httpBasic": {

              "username": "xxx",

              "password": "yyy"

            }

          }

        }

      },

      "outputs": [

        {

          "schema": "iglu:com.acme/user/jsonschema/1-0-0",

          "json": {

            "jsonPath": "$.record"

          }

        }

      ],

      "cache": {

        "size": 3000,

        "ttl": 60

      },

      "ignoreOnError": false

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### `inputs`

The enrichment can use any property in the event as input data source. The input data can be extracted from:

- [Atomic event properties](/docs/fundamentals/canonical-event/) such as `user_id`
- [Self-describing event](/docs/fundamentals/events/#self-describing-events) fields
- [Entities](/docs/fundamentals/entities/) attached by tracker SDKs
- Entities attached by other enrichments

The custom API enrichment runs after most other enrichments, so it can access data added by them. Only the IP anonymization and PII pseudonymization enrichments run after the custom API enrichment.

Specify an array of `inputs` to use as keys when performing your API lookup. Each input consists of a `key` and a source: `pojo` for atomic event fields, or `json` for JSON fields, whether event or entity.

Key names can contain only alphanumeric symbols, hyphens, and underscores.

For `json`, specify the field name as either `unstruct_event` for self-describing event fields, `contexts` for fields in entities added during tracking, or `derived_contexts` for fields in enrichment entities. Add two additional fields:

- `schemaCriterion` is the Iglu schema URI. You can specify all versions of the schema (`*-*-*`), or a specific major version (e.g. `1-*-*`), major plus minor (e.g. `1-1-*`) or a full major-minor-patch version (e.g. `1-1-1`)
- `jsonPath` is the [JSON Path statement](http://goessner.net/articles/JsonPath/) to navigate to the field inside the JSON that you want to use as the input.

The resolved values should be primitive types (string, number, or boolean).

### `api`

Configure the API access with `api`. The enrichment supports `GET`, `POST`, and `PUT` methods, and both HTTP and HTTPS protocols.

For the `uri` field, specify the full URI, including the protocol. You can attach a querystring to the end of the URI. You can also embed the keys from your `inputs` section in the URI, by wrapping the key in `{{}}` brackets:

```json
"uri": "http://api.acme.com/users/{{client}}/{{user}}?format=json"
```

If a key required in the `uri` wasn't found in any of the `inputs`, then the lookup won't proceed, but this will **not** be flagged as a failure.

The only supported `authentication` option is `httpBasic`. Provide a `username` and/or `password` for the enrichment to connect to your API using [basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). Some APIs use only the `username` or `password` field to contain an API key; in this case, set the other property to an empty string `""`.

If your API is unsecured (for example, because it's only accessible from inside your private subnet, or because you're using an IP address allowlist) then configure the `authentication` section like this:

```json
"authentication": { }
```

### `outputs`

This enrichment assumes that your API returns a JSON object containing one or more properties that you want to add to your event. You'll need to specify the schema or data structure that the enrichment should use to define the retrieved data.

Each entry in the `outputs` array needs two fields:

- `schema` to specify the schema URI you want to attach to the event.
- `json` and `jsonPath` to specify which part of the returned JSON you want to add to the enriched event. Use `$` if you want to attach the returned JSON as is.

The `outputs` array must have at least one entry in it.

If the JSON path specified can't be found within the API response, then the lookup and the event will be flagged as a failure - unless `ignoreOnError` is set to `true`.

### `cache`

An enrichment can run many millions of time per hour, effectively launching a DoS attack on a data source. The `cache` configuration attempts to minimize the number of lookups performed.

The cache is an LRU (least-recently used) cache, where less frequently accessed values are evicted to make space for new values. The `uri` with all keys populated is used as the key in the cache. Configure the `cache` as follows:

- `size` is the maximum number of entries to hold in the cache at any one time. The minimum value is `1`.
- `ttl` is the number of seconds that an entry can stay in the cache before it is forcibly evicted. This is useful to prevent stale values from being retrieved in the case that your API can return different values for the same key over time.

### `ignoreOnError`

When set to `true`, if the enrichment fails for any reason, the event is still considered successfully enriched. It'll be loaded as usual, except without the entities added by the enrichment.

When set to `false`, the event will become a [failed event](/docs/fundamentals/failed-events/) if the API call fails.

## Edge case handling

This enrichment can use any third-party RESTful service to fetch data in JSON format. In most cases, we recommend using your own private server to maintain performance. Third-party services could cause slowdown of your enrichment process.

This table describes what will happen under different conditions:

| Scenario                                                                      | Outcome                                                                                                                                                    |
| ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A provided JSONPath is invalid.                                               | Failed event, unless `ignoreOnError` is set to `true`.                                                                                                     |
| Any one of the input keys wasn't found.                                       | The HTTP request won't be sent, and no entities will be added. The event will otherwise be processed as usual - not failed.                                |
| More than one entity in the event matches the `schemaCriterion`.              | The first matching entity found will be used.                                                                                                              |
| Multiple inputs share the same key (don't do this).                           | The last input configured will be picked.                                                                                                                  |
| An input JSONPath matches a non-primitive value.                              | The enrichment will try to stringify it, likely resulting in an invalid URL. If so, it will cause a failed event, unless `ignoreOnError` is set to `true`. |
| The output's JSONPath wasn't found.                                           | Failed event, unless `ignoreOnError` is set to `true`.                                                                                                     |
| The response returned JSON which is not valid according to the output schema. | Failed event, unless `ignoreOnError` is set to `true`.                                                                                                     |
| The server returned any non-successful response or timed-out.                 | Failed event, unless `ignoreOnError` is set to `true`.                                                                                                     |

## Output

This enrichment adds entities based on your configuration.

Example API response:

```json
// GET http://api.acme.com/users/northwind-traders/123?format=json

{

  "metadata": {

    "whenCreated": 1448371243,

    "whenUpdated": 1448373431

  },

  "record": {

    "name": "Bob Thorpe",

    "id": "123"

  }

}
```

With this configuration:

```json
"outputs": [ {

  "schema": "iglu:com.acme/user/jsonschema/1-0-0",

  "json": {

    "jsonPath": "$.record"

  }

} ]
```

The enrichment will add this entity to your event:

```json
{

  "schema": "iglu:com.acme/user/jsonschema/1-0-0",

  "data": {

    "name": "Bob Thorpe",

    "id": "123"

  }

}
```

---

# Custom JavaScript enrichment
> Write custom JavaScript code to transform and enrich events with flexible data manipulation logic.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/

With this enrichment, you can write a JavaScript function to be executed for each event. Use this enrichment to apply your own business logic to your events, including:

- Adding extra data to the event in the form of [entities](/docs/fundamentals/entities/)
- Modifying event fields directly
- Discarding the event so that it goes to [failed events](/docs/fundamentals/failed-events/) rather than your data destination

## Requirements

Key considerations when writing your own JavaScript enrichment:

- Make sure your code works for _all_ of your events, not just the particular types of events you're interested in. Unhandled exceptions will result in [failed events](/docs/fundamentals/failed-events/).
- Don't try to share state across multiple enriched events. Enrichments run inside a highly parallel application with multiple independent instances, so this won't work.
- In your enrichment code, avoid CPU-intensive tasks such as encryption, or IO-intensive tasks such as requests to an external service, without thoroughly benchmarking the impact they might have on your event processing time.
- The enrichment code has access to the Java standard library, and therefore to the filesystem of the machine it's running on. Proceed with caution when copying code from untrusted sources. Snowplow CDI Cloud customers don't have access to Java methods, for security reasons.

Your JavaScript enrichment code should contain a function called `process`:

- This function will receive each event as its first argument
- It can optionally return an array of [entities](/docs/fundamentals/entities/) to be added to the event
- Any uncaught exceptions will result in [failed events](/docs/fundamentals/failed-events/)

```js
function process(event) {

  // do something with the event...

  ...



  // add entities to the event

  return [ ... ];

}
```

You can split more complex logic into multiple helper functions and variables as you see fit, as long as you comply with this interface.

You can only have _one_ JavaScript enrichment, and hence a single `process` function for your pipeline.

> **Tip:** The JavaScript enrichment uses the standalone [Nashorn engine](https://github.com/openjdk/nashorn) with ES6 support.
>
> Not all ES6 features are guaranteed to work. If you are unsure whether a specific feature is available, [test it in a local environment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/) before deploying.

## Configuration

The enrichment takes these parameters:

| Parameter | Required | Description                           |
| --------- | -------- | ------------------------------------- |
| `script`  | ✅        | The JavaScript code to run.           |
| `config`  | ❌        | Parameters to pass to the enrichment. |

**Console:**

Console has two editors: a JavaScript editor for the `script`, and a JSON editor for any `config` parameters. Enter your `script` directly as JavaScript — no base64 encoding required. For example:

```js
function process(event) {



  var platform = event.getPlatform(),

      appId    = event.getApp_id();



  if (platform == "server" && appId != "secret") {

    throw "Server-side event has invalid app_id: " + appId;

  }



  if (appId == null) {

    return [];

  }



  var appIdUpper = new String(appId.toUpperCase());



  return [ { schema: "iglu:com.acme/foo/jsonschema/1-0-0",

               data: { appIdUpper: appIdUpper }

           } ];

}
```

```json
{}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/javascript_script_config/jsonschema/1-0-0",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow",

    "name": "javascript_script_config",

    "enabled": false,

    "parameters": {

      "script": "ZnVuY3Rpb24gcHJvY2VzcyhldmVudCkgew0KDQogIHZhciBwbGF0Zm9ybSA9IGV2ZW50LmdldFBsYXRmb3JtKCksDQogICAgICBhcHBJZCAgICA9IGV2ZW50LmdldEFwcF9pZCgpOw0KDQogIGlmIChwbGF0Zm9ybSA9PSAic2VydmVyIiAmJiBhcHBJZCAhPSAic2VjcmV0Iikgew0KICAgIHRocm93ICJTZXJ2ZXItc2lkZSBldmVudCBoYXMgaW52YWxpZCBhcHBfaWQ6ICIgKyBhcHBJZDsNCiAgfQ0KICANCiAgaWYgKGFwcElkID09IG51bGwpIHsNCiAgICByZXR1cm4gW107DQogIH0NCg0KICAvLyBVc2UgbmV3IFN0cmluZygpIGJlY2F1c2UgaHR0cDovL25lbHNvbndlbGxzLm5ldC8yMDEyLzAyL2pzb24tc3RyaW5naWZ5LXdpdGgtbWFwcGVkLXZhcmlhYmxlcy8NCiAgdmFyIGFwcElkVXBwZXIgPSBuZXcgU3RyaW5nKGFwcElkLnRvVXBwZXJDYXNlKCkpOw0KDQogIHJldHVybiBbIHsgc2NoZW1hOiAiaWdsdTpjb20uYWNtZS9mb28vanNvbnNjaGVtYS8xLTAtMCIsDQogICAgICAgICAgICAgICBkYXRhOiB7IGFwcElkVXBwZXI6IGFwcElkVXBwZXIgfQ0KICAgICAgICAgICB9IF07DQp9"

    }

  }

}
```

***

> **Note:** If you're using [Snowplow Console](https://console.snowplowanalytics.com) toconfigure the enrichment, or configuring or [Micro](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/), provide your script directly as JavaScript.
>
> For other configurations, you'll need to provide the JavaScript code encoded in base64.

## Available properties and methods

Your JavaScript enrichment code has access to the entire event payload, including request headers, plus any parameters you define.

### Atomic fields

Each of the [atomic event fields](/docs/fundamentals/canonical-event/) has its own getter. The name is the capitalized field name prepended with `get`, for example `event.getUser_ipaddress()` or `event.getGeo_country()`.

This example shows how to get the `app_id` field:

```js
function process(event) {

  const appId = event.getApp_id();



  ...
```

> **Note:** One exception is `refr_device_tstamp`, where the getter method is called `getRefr_dvce_tstamp` and _not_ `getRefr_device_tstamp`.

### Event properties

To access [self-describing event](/docs/fundamentals/events/#self-describing-events) fields, use `event.getUnstruct_event()`. This will return the self-describing event as a JSON string. Parse it to access the schema at `.data.schema` and the individual fields at `.data.data`.

```js
function process(event) {

  ...



  const myEvent = JSON.parse(event.getUnstruct_event());



  if (myEvent) {

    // the event schema

    const mySchema = myEvent.data.schema;



    // the event data

    const myData = myEvent.data.data;



    ...
```

For [baked-in events](/docs/fundamentals/events/#baked-in-events) with no schema, `getUnstruct_event()` will return `null`. The pattern here still works, as `null` is a valid input for `JSON.parse`.

### Entity properties

Access any tracker-defined entities via `event.getContexts()`. Parse the returned JSON string to access the individual entities in the array.

For entities added by other enrichments, use `event.getDerived_contexts()` instead. This feature is supported since Enrich 3.8.0 (and Snowplow Micro 1.7.1). In prior versions, this function always returns `null`.

```js
function process(event) {

  ...



  const trackerEntities = JSON.parse(event.getContexts());



  if (trackerEntities) {

    // loop through the entities

    for (const entity of trackerEntities.data) {

      if (entity.schema.startsWith('iglu:org.my-company/my-schema/jsonschema/1')) {

        // work with the entity

        const myField = entity.data.myField;



        ...

      }

    }

  }



  ...
```

For events with no entities attached, `getContexts()` will return `null`. The pattern above works, as `null` is a valid input for `JSON.parse`.

### Passed parameters

Use the `config` configuration property to pass an object of parameters to the enrichment. This can be useful when you want to quickly reconfigure the enrichment without updating the JavaScript code.

For example, to pass `"foo": 3` and `"nested": {"bar": "test"}`:

```json
{

	"schema": "iglu:com.snowplowanalytics.snowplow/javascript_script_config/jsonschema/1-0-1",

	"data": {

		"vendor": "com.snowplowanalytics.snowplow",

		"name": "javascript_script_config",

		"enabled": true,

		"parameters": {

			"script": "script in base64",

			"config": {

				"foo": 3,

				"nested": {

					"bar": "test"

				}

			}

		}

	}

}
```

Access your provided parameters using the second argument of the `process` function. For example, to set the `app_id` field to the value of the `nested.bar` parameter:

```js
function process(event, params) {

  event.setApp_id(params.nested.bar);

}
```

If you don't provide any parameters, the second argument will be an empty object `{}`.

This feature is available since Enrich 4.1.0.

### HTTP request headers

Your script can access the HTTP headers that came with the original request to the Collector. This can be useful to apply different logic to events depending on the presence or value of certain headers.

Access the array of headers in the third argument to the `process` function. Each header is a string in the format `HEADER:value`.

For example, to pass a token via a custom `X-JWT` header and then use it in the enrichment:

```js
function process(event, params, headers) {

  for (header of headers) {

    const token = header.match(/X-JWT:(.+)/i)

    if (token) {

      const value = token[1].trim()

      // do something with the value

    }

  }

}
```

> **Tip:** Headers aren't case sensitive, so we recommend using a case insensitive match, as shown in the example.
>
> It's also good practice to trim whitespace from the value, as there could be some space around the `:` character.

Accessing request headers is supported since Enrich 5.1.1.

### Java methods

> **Note:** For security reasons, this feature isn't available to [Snowplow CDI Cloud](/docs/get-started/) customers.

Because the JavaScript enrichment runs inside the Enrich application, it has access to the Java standard library, as well as the Java libraries used by Enrich. You can call Java methods via their fully qualified path, for example:

```js
function process(event) {

  ...



  const salt = 'pepper';

  const hashedIp = org.apache.commons.codec.digest.DigestUtils.sha256Hex(event.getUser_ipaddress() + salt);

  ...

}
```

This example shows how to hash a user's IP address. You could do this automatically using the [PII pseudonymization enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/).

## Edit properties

Your script can modify any of the event fields. The old value won't be stored or saved at all. If you want to preserve the original value, consider adding a new entity instead.

Editing fields can be particularly useful when you want to apply logic based on the value of another field.

### Atomic fields

Each of the [atomic fields](/docs/fundamentals/canonical-event/) has its own setter method, with the name being the capitalized field name prepended with `set`, for example `setApp_id('myAppId')` or `setGeo_country('US')`.

You can use this to edit an existing value, to set a value that is currently `null`, or to erase a field.

```js
function process(event) {

  event.setMkt_source('Facegoog');



  if (event.getGeo_country() === 'US') {

    event.setGeo_latitude(null);

    event.setGeo_longitude(null);

  }



  ...
```

> **Note:** One exception is `refr_device_tstamp`, where the setter method is called `setRefr_dvce_tstamp` and _not_ `setRefr_device_tstamp`.

### Event properties

To modify fields of a self-describing event, you'll need to unpack them, make the modification, and then set the whole modified object using `setUnstruct_event`.

For example, to update a field called `myField` inside the event:

```js
function process(event) {

  ...



  // unpack the self-describing event

  const myEvent = JSON.parse(event.getUnstruct_event());



  if (myEvent && myEvent.data.schema === ...) {

    // update a field inside

    myEvent.data.data.myField = 'new value';



    // pack the self-describing event back

    event.setUnstruct_event(JSON.stringify(myEvent));

  }



  ...
```

### Entity properties

To modify fields of an entity, you'll need to unpack them, make the modification, and then set the whole modified object using `setContexts`. You can only edit fields in tracker-set entities with this method.

You can also use this approach to delete tracker entities, by filtering them out from the array.

For example, to delete unwanted entities and update a field called `myField` inside an entity:

```js
function process(event) {

  ...



  // unpack the entities

  const entities = JSON.parse(event.getContexts());



  if (entities) {

    // remove entities with a specific schema

    entities.data = entities.data.filter(e => e.schema !== 'iglu:com.my-company/unwanted/jsonschema/1-0-0');



    // loop through the remaining entities

    for (const entity of entities.data) {

      if (entity.schema === ...) {

        // update a field inside

        entity.data.myField = entity.data.myField + 1;

      }

    }



    // pack the entities back

    event.setContexts(JSON.stringify(entities));

  }



  ...
```

For entities added by enrichments, see the [Delete entities](#delete-entities) section on this page for details of how to edit or delete them.

## Add or delete entities

Your script can add new entities to the event, or delete existing entities.

### Add entities

The optional return value of the `process` function is an array of extra entities to add to the event.

In this example, the events are given an entity that distinguishes traffic from internal and external systems:

```js
const internalIPs = new Set([

  "1.2.3.4",

  "2.3.4.5"

]);



function process(event) {

  const ip = event.getUser_ipaddress();

  const traffic_source = internalIPs.has(ip) ? 'internal' : 'external';



  return [{

    schema: "iglu:com.my-company/traffic_source/jsonschema/1-0-0",

    data: { traffic_source }

  }];

}
```

Check out the [testing guide](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/#iterating-on-code-and-schemas) for advice on iterating on your schema and JavaScript script together.

> **Warning:** Make sure that the schemas of your entities are defined and accessible to your pipeline. If the enrichment can't find a schema, this will cause a [failed event](/docs/fundamentals/failed-events/).

Your array of entities will be passed to `JSON.stringify()` before being attached to the event. If your new entity has a `NaN` value, it will become `null`. If it has an `undefined` value, it will be dropped. If it has circular references, an exception will be thrown and the event will become a failed event.

### Delete entities

To delete tracker-set entities, use the same approach as for editing them.

To edit or delete entities added by enrichments, use `getDerived_contexts()` with `eraseDerived_contexts()`. The `eraseDerived_contexts()` method will delete all the enrichment entities from the event. You can then return a modified array of entities to replace them:

```js
function process(event) {

  const enrichmentEntities = JSON.parse(event.getDerived_contexts())



  // delete the existing enrichment entities array from the event

  event.eraseDerived_contexts()



  // modify the entities

  for (const entity of enrichmentEntities.data) {

    if (entity.schema === ...) {

      // update a field inside

      entity.data.myField = entity.data.myField + 1

    }

  }



  // remove entities with a specific schema

  const filtered = enrichmentEntities.data.filter(e => e.schema !== 'iglu:com.my-company/unwanted/jsonschema/1-0-0')



  // returned the updated array of enrichment entities

  // this will replace the original one

  return filtered

}
```

The `eraseDerived_contexts()` method is available since Enrich 5.4.0.

## Discard events

Sometimes you don't want the event to appear in your data warehouse or lake, e.g. because you suspect it comes from a bot and not a real user.

### Drop completely

You can also drop any event, using `event.drop()`. This is a destructive operation: dropped events won't be stored or sent to any stream or destination.

For example, to drop events with a user agent containing "Googlebot":

```js
const botPattern = /.*Googlebot.*/;



function process(event) {

  const useragent = event.getUseragent();



  if (useragent !== null && botPattern.test(useragent)) {

    event.drop();

  }

}
```

> **Warning:** There's no way to recover dropped events.

This feature is available since Enrich 5.3.0.

### Cause a failed event

You can send an event to [failed events](/docs/fundamentals/failed-events/) by deliberately throwing an unhandled exception.

This will create an "enrichment failure" failed event, which may be tricky to distinguish from genuine failures in your enrichment code.

For example, to filter out events with a user agent containing "Googlebot":

```js
const botPattern = /.*Googlebot.*/;



function process(event) {

  const useragent = event.getUseragent();



  if (useragent !== null && botPattern.test(useragent)) {

    throw "Filtered event produced by Googlebot";

  }

}
```

The [bot detection enrichment](/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/) consolidates signals from multiple enrichments into a single entity. This example uses the `bot_detection` entity to send bot events to failed events:

```js
function process(event) {

  const entities = JSON.parse(event.getDerived_contexts());

  if (entities) {

    for (const entity of entities.data) {

      if (entity.schema.startsWith('iglu:com.snowplowanalytics.snowplow/bot_detection/jsonschema/1') &&

          entity.data.bot) {

        throw "Filtered a spider/bot event";

      }

    }

  }

}
```

---

# Test JavaScript enrichments locally
> Test JavaScript enrichment code locally using Node.js before deploying to production pipelines.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/

You can (and should!) test your enrichment with [Snowplow Micro](/docs/testing/snowplow-micro/) before adding it to your production pipeline.

## Basic setup

While it's possible to use Micro [through Console](/docs/testing/snowplow-micro/console/), running it locally has the benefit that you can add logging statements and look at their output.

If you haven't run Micro on your machine before, take a look at the [usage guide](/docs/testing/snowplow-micro/local/). Adding the JavaScript enrichment to Micro is similar to [adding other enrichments](/docs/testing/snowplow-micro/local/enrichments/).

> **Tip:** Testing `.js` files directlyWhile you can add the full enrichment configuration in a JSON file (with your JavaScript encoded in base64), you can also just drop a `.js` file in your enrichments folder. Micro will interpret it as JavaScript enrichment code. This way, you don’t need to keep converting your enrichment to base64 as you iterate on it. _(Make sure to either use a `.json` or a `.js` file, but not both.)_

For example, if your enrichment code is in `script.js` in the current directory, you can execute the following command to run it in Micro:

```bash
docker run -p 9090:9090 \

--mount type=bind,source=$(pwd)/script.js,destination=/config/enrichments/script.js \

snowplow/snowplow-micro:4.4.0
```

Next, point your tracking code to `localhost:9090` (see the [usage guide](/docs/testing/snowplow-micro/local/#sending-events-to-micro) for more details). Now the events you send to Micro will go through your enrichment code.

## Debugging

When debugging a JavaScript enrichment with Micro, there are a few sources of useful information: the [dashboard](/docs/testing/snowplow-micro/ui/) (available since Micro 2.0.0), the logs and the [REST API](/docs/api-reference/snowplow-micro/api/).

### Dashboard

The dashboard will show you which events are enriched successfully and which fail.

For each failed event, you can also click on the failure badge to look at the detailed failure message.

### Logs

The logs will quickly show you any failure. For example, if you forgot to add your schema:

```text
[WARN] EventLog - BAD {

  "schemaKey" : "iglu:com.my-company/my-schema/jsonschema/1-0-0",

  "error" : {

    "error" : "ResolutionError",

    ...
```

Or if you made a typo in a function name:

```text
[WARN] EventLog - BAD {

  "error" : "Error during execution of JavaScript function: [TypeError: event.getLongtitude is not a function in <eval> at line number 2]"

}
```

> **Note:** If you `throw` in your JavaScript code to [discard the event](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/#discard-events), you will get a similar failure message in the logs, but it would be expected.

### Tracing

You can also add temporary `print` statements to your code to inspect certain values.

> **Tip:** Make sure to remove all debug printing when adding the enrichment to a production pipeline.

Here’s an example:

```js
print(`--- URL is: ${event.getPage_url()}`);
```

This will be printed directly to the output from Micro (in the terminal — currently this information is not available in the Micro dashboard).

> **Info:** There is no `console.log`Note that `console.log` and `console.dir` familiar to JavaScript developers _will not_ work in this context.

If you want to pretty-print an object or array, the best approach is to use `JSON.stringify`:

```js
const myEvent = JSON.parse(event.getUnstruct_event());

print(JSON.stringify(myEvent.data, null, 2));
```

### REST API

Using Micro’s [REST API](/docs/api-reference/snowplow-micro/api/), you can verify that your code modifies event fields or adds entities as expected. The most useful endpoint is `/micro/good`, which will return all successfully processed events:

```bash
curl localhost:9090/micro/good
```

For example, if your code adds some entities, you will find them in the output from the REST API, under `derived_contexts` field for each event:

```json
...

"derived_contexts": {

  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",

  "data": [

    {

      "schema": "...",

      "data": { ... }

    },

    ...

  ]

}

...
```

## Iterating on code and schemas

If your enrichment [adds extra entities to the event](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/#add-entities), you might be iterating on the schemas for those entities at the same time you are iterating on your enrichment code.

You can simplify your development and testing flow by [adding schemas to Micro](/docs/testing/snowplow-micro/local/schemas/#adding-schemas-directly-to-micro) as local files. For example, your current directory might look like this:

```text
script.js

schemas

└── com.my-company

    └── my-schema

        └── jsonschema

            └── 1-0-0
```

This command will add both the script and the schema to Micro:

```bash
docker run -p 9090:9090 \

--mount type=bind,source=$(pwd)/schemas,destination=/config/iglu-client-embedded/schemas \

--mount type=bind,source=$(pwd)/script.js,destination=/config/enrichments/script.js \

snowplow/snowplow-micro:4.4.0
```

## Testing with more complete events

Here are a few more tips on getting the most out of Micro when testing your JavaScript enrichment.

### Enriched fields

If your enrichment uses fields filled by other enrichments, you will want to [enable those](/docs/testing/snowplow-micro/local/enrichments/) as well.

### IP addresses

If your enrichment relies on the `user_ipaddress` field or the `geo_*` fields, with the default setup you will discover that `user_ipaddress` is your local one (e.g. something like `192.168.0.42`). Subsequently, the `geo_*` fields are all `null`. That’s because you have both Micro and your tracking code running locally on the same machine.

If you are using a web browser to test your site or app, you can spoof a specific IP address using a browser plugin that sets an `X-Forwarded-For` header. For example, here are plugins for [Chrome](https://chrome.google.com/webstore/detail/x-forwarded-for-header/hkghghbnihliadkabmlcmcgmffllglin/) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/x-forwarded-for-injector/). Install the plugin and set the IP address to your liking.

Alternatively, check out the section on [exposing Micro via a public domain name](/docs/testing/snowplow-micro/local/remote-usage/#exposing-micro-via-a-public-domain-name) — with this approach you can get a public URL for your Micro, to which you can point your tracking code. Now the interaction between your tracking and your Micro will go through the internet, and you will get a realistic IP address in your events.

### Cookies

If your enrichment relies on cookies, you may find it difficult to reproduce all the cookies set by your website or app in a local setup.

For a solution, check out the section on [locally resolving an existing domain name to Micro](/docs/testing/snowplow-micro/local/remote-usage/#locally-resolving-an-existing-domain-name-to-micro). With this approach you can hook into an existing website or app, receiving all cookies in your Micro.

> **Tip:** If the values of cookie-based fields (e.g. `network_userid`) are not what you expect, make sure you [configure Micro](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-collector-and-enrich-configuration) to use the same cookie name as your website or app (the default is `micro`). For example, to set it to `sp`:
>
> ```bash
> docker run ... \
>
> snowplow/snowplow-micro:4.4.0 \
>
> -Dcollector.cookie.name=sp
> ```

---

# Custom SQL query enrichment
> Query relational databases during enrichment to attach lookup data from MySQL, PostgreSQL, or other SQL databases.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/

The SQL query enrichment lets you add data to a Snowplow event via your own MySQL or PostgreSQL relational database.

Supported database types:

- MySQL, plus variants which speak MySQL e.g. MariaDB, Amazon Aurora
- PostgreSQL, plus variants which speak PostgreSQL

We don't recommend using this enrichment with analytical databases such as Redshift. These databases are optimized for large analytical queries, not the high-frequency, low-latency per-event lookups this enrichment requires.

## Configuration

For historical reasons, the configuration uses terms that are no longer used elsewhere in Snowplow.

The enrichment takes these parameters:

| Parameter       | Required | Description                                                          |
| --------------- | -------- | -------------------------------------------------------------------- |
| `inputs`        | ✅        | Event values to substitute in SQL placeholders.                      |
| `database`      | ✅        | Defines access to the database.                                      |
| `query`         | ✅        | The SQL statement to query your database.                            |
| `output`        | ✅        | How to convert the returned rows into self-describing JSON entities. |
| `cache`         | ✅        | Whether to store retrieved rows.                                     |
| `ignoreOnError` | ❌        | Whether to make the event fail if the API request fails.             |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "inputs": [

    {

      "placeholder": 1,

      "pojo": {

        "field": "user_id"

      }

    },

    {

      "placeholder": 1,

      "json": {

        "field": "contexts",

        "schemaCriterion": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",

        "jsonPath": "$.userId"

      }

    },

    {

      "placeholder": 2,

      "pojo": {

        "field": "app_id"

      }

    }

  ],

  "database": {

    "postgresql": {

      "host": "cluster01.redshift.acme.com",

      "port": 5439,

      "sslMode": true,

      "username": "snowplow_enrich_ro",

      "password": "1asIkJed",

      "database": "crm"

    }

  },

  "query": {

    "sql": "SELECT username, email_address, date_of_birth FROM tbl_users WHERE user = ? AND client = ? LIMIT 1"

  },

  "output": {

    "expectedRows": "AT_MOST_ONE",

    "json": {

      "schema": "iglu:com.acme/user/jsonschema/1-0-0",

      "describes": "ALL_ROWS",

      "propertyNames": "CAMEL_CASE"

    }

  },

  "cache": {

    "size": 3000,

    "ttl": 60

  },

  "ignoreOnError": false

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/sql_query_enrichment_config/jsonschema/1-0-1",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "name": "sql_query_enrichment_config",

    "enabled": false,

    "parameters": {

      "inputs": [

        {

          "placeholder": 1,

          "pojo": {

            "field": "user_id"

          }

        },

        {

          "placeholder": 1,

          "json": {

            "field": "contexts",

            "schemaCriterion": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",

            "jsonPath": "$.userId"

          }

        },

        {

          "placeholder": 2,

          "pojo": {

            "field": "app_id"

          }

        }

      ],

      "database": {

        "postgresql": {

          "host": "cluster01.redshift.acme.com",

          "port": 5439,

          "sslMode": true,

          "username": "snowplow_enrich_ro",

          "password": "1asIkJed",

          "database": "crm"

        }

      },

      "query": {

        "sql": "SELECT username, email_address, date_of_birth FROM tbl_users WHERE user = ? AND client = ? LIMIT 1"

      },

      "output": {

        "expectedRows": "AT_MOST_ONE",

        "json": {

          "schema": "iglu:com.acme/user/jsonschema/1-0-0",

          "describes": "ALL_ROWS",

          "propertyNames": "CAMEL_CASE"

        }

      },

      "cache": {

        "size": 3000,

        "ttl": 60

      },

      "ignoreOnError": false

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### `inputs`

The SQL statement you provide can include placeholders. These will be replaced with actual values from the event, called `inputs`.

The enrichment can use any property in the event as input data. The values can be extracted from:

- [Atomic event properties](/docs/fundamentals/canonical-event/) such as `user_id`
- [Self-describing event](/docs/fundamentals/events/#self-describing-events) fields
- [Entities](/docs/fundamentals/entities/) attached by tracker SDKs
- Entities attached by other enrichments

Each input consists of a `placeholder` index and a source: `pojo` for atomic event fields, or `json` for JSON fields, whether event or entity.

The `placeholder` is the index of the `?` SQL placeholder which this input will populate. It must be an integer greater than or equal to 1.

For `json`, specify the field name as either `unstruct_event` for self-describing event fields, `contexts` for fields in entities added during tracking, or `derived_contexts` for fields in enrichment entities. Add two additional fields:

- `schemaCriterion` is the Iglu schema URI. You can specify all versions of the schema (`*-*-*`), or a specific major version (e.g. `1-*-*`), major plus minor (e.g. `1-1-*`) or a full major-minor-patch version (e.g. `1-1-1`)
- `jsonPath` is the [JSON Path statement](http://goessner.net/articles/JsonPath/) to navigate to the field inside the JSON that you want to use as the input.

The resolved values should be primitive types (string, number, or boolean).

As shown in the example configuration, you can have multiple inputs with the same placeholder index. The enrichment will use the last configured input for that index that resolves to a non-null value. This allows you to specify fallback inputs.

### `database`

Configure how the enrichment should access your database in the `database` section. Populate all properties in the `postgresql` or `mysql` configuration object:

- `host`: the hostname or IP address of the database server or cluster
- `port`: the port to connect to the database on
- `sslMode`: whether the database requires connections to use SSL
- `username`: the username for the database login
- `password`: the password for this username
- `database`: the name of the specific database to run the query against

> **Tip:** We strongly recommend restricting the username to minimal read-only permissions on just the tables required to execute the SQL query.

If your database server has additional authentication in place, such as IP allowlisting, configure it to permit access from the NAT gateway associated with the Snowplow pipeline.

### `query`

Consists of a single `sql` key with a SQL query in the form of prepared statement. It will run on your relational database to return rows to attach to this event as entities. Use `?` placeholders in the query to indicate where the input values should be substituted. Values will be inserted according to their type: strings will be quoted, numbers won’t.

If a placeholder index required in the `sql` prepared statement isn't found in any of the `inputs`, then the lookup won't proceed, but this will **not** be flagged as a failure.

A final `;` is optional.

> **Warning:** This enrichment makes no attempt to sanitize the SQL statement, nor to verify that the SQL statement doesn't have harmful side effects such as SQL injection.

### `output`

The enrichment adds the returned rows to the event as one or more entities. You'll need to specify the schema or data structure that the enrichment should use to define the retrieved data.

The `output` object has two keys: `expectedRows` and `json`. The `expectedRows` enum defines the expected SQL output:

- `EXACTLY_ONE`: exactly one row is expected. 0 or 2+ rows will throw an error, causing the entire event to fail processing.
- `AT_MOST_ONE`: either one or zero rows is expected. 2+ rows will throw an error.
- `AT_LEAST_ZERO`: between 0 and N rows are expected, i.e. an array of results.
- `AT_LEAST_ONE`: between 1 and N rows are expected, i.e. an array of results. 0 rows will throw an error.

Within the `json` key, use `schema` to specify the URI of the schema you want to attach to the event.

The `describes` field configures whether to add one entity containing all the returned rows, or one entity per returned row. This will depend on your schema definition. Supported options are:

- `ALL_ROWS` means that the added entity will contain all the returned rows
- `EVERY_ROW` means that one entity will be added per returned row

The `propertyNames` field supports reformatting of the returned columns to fit the schema. Supported options are:

- `AS_IS`: preserve the column names exactly as they are
- `CAMEL_CASE`: e.g. `date_of_birth` becomes `dateOfBirth`
- `PASCAL_CASE`: e.g. `date_of_birth` becomes `DateOfBirth`
- `SNAKE_CASE`: e.g. `dateOfBirth` becomes `date_of_birth`
- `LOWER_CASE`: changes all characters to lowercase
- `UPPER_CASE`: changes all characters to uppercase

You could also use column aliasing in your SQL statement to change individual column names.

### `cache`

An enrichment can run many millions of time per hour, effectively launching a DoS attack on a data source. The `cache` configuration attempts to minimize the number of lookups performed.

The cache is an LRU (least-recently used) cache, where less frequently accessed values are evicted to make space for new values. Configure the `cache` as follows:

- `size` is the maximum number of entries to hold in the cache at any one time. The minimum value is `1`.
- `ttl` is the number of seconds that an entry can stay in the cache before it is forcibly evicted. This is useful to prevent stale values from being retrieved in the case that your database can return different values for the same key over time.

### `ignoreOnError`

When set to `true`, if the enrichment fails for any reason, the event is still considered successfully enriched. It'll be loaded as usual, except without the entities added by the enrichment.

When set to `false`, the event will become a [failed event](/docs/fundamentals/failed-events/) if the SQL query fails.

## Edge case handling

This enrichment uses a relational database to fetch data in JSON format. This table describes what will happen under different conditions:

| Scenario                                                         | Outcome                                                                                                                              |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| A provided JSONPath is invalid.                                  | Failed event, unless `ignoreOnError` is set to `true`.                                                                               |
| More than one entity in the event matches the `schemaCriterion`. | The first matching entity is used.                                                                                                   |
| Multiple inputs share the same placeholder index.                | The last configured input is used.                                                                                                   |
| Any placeholder index has no value.                              | The SQL query won't run and no entities will be added. The event is otherwise processed as usual — not failed.                       |
| An input value is non-primitive (object or array), or `null`.    | Treated as not found. The SQL query won't run and no entities will be added. The event is otherwise processed as usual — not failed. |
| The database query failed for any reason, or timed out.          | Failed event, unless `ignoreOnError` is set to `true`.                                                                               |
| The database connection was lost.                                | Failed event, unless `ignoreOnError` is set to `true`.                                                                               |

If the database connection is lost, the enrichment will attempt to reestablish it on the next event.

## Output

This enrichment adds entities based on your configuration.

### Single row

With this configuration:

```json
...

  "query": {

    "sql": "SELECT username, date_of_birth FROM tbl_users WHERE user = ?"

  },

  "output": {

    "expectedRows": "AT_MOST_ONE",

    "json": {

      "schema": "iglu:com.acme/user/jsonschema/1-0-0",

      "describes": "ALL_ROWS",

      "propertyNames": "CAMEL_CASE"

    }

  },

...
```

And this query result:

```sql
SELECT username, date_of_birth FROM tbl_users WHERE user = 123;

| username | date_of_birth |

| -------- | ------------- |

| karl     | 1980-06-12    |
```

The enrichment would add this entity to your event:

```json
{

  "schema": "iglu:com.acme/user/jsonschema/1-0-0",

  "data": {

    "username": "karl",

    "dateOfBirth": "1980-06-12T00:00:00"

  }

}
```

If the query returned no rows, e.g.:

```sql
SELECT username, date_of_birth FROM tbl_users WHERE user = 123;

| username | date_of_birth |

| -------- | ------------- |
```

The enrichment won't add an entity, but the event would continue processing.

If the query returned more than one row, e.g.:

```sql
SELECT username, date_of_birth FROM tbl_users WHERE user = 123;

| username | date_of_birth |

| -------- | ------------- |

| karl     | 1980-06-12    |

| mary     | 1975-03-22    |
```

An error would be triggered, on account of the `AT_MOST_ONE` setting, and the event would become a failed event.

### Multiple rows

With this `ALL_ROWS` configuration:

```json
...

  "query": {

    "sql": "SELECT * FROM product WHERE category = ?"

  },

  "output": {

    "expectedRows": "AT_LEAST_ZERO",

    "json": {

      "schema": "iglu:com.acme/products/jsonschema/1-0-0",

      "describes": "ALL_ROWS",

      "propertyNames": "AS_IS"

    }

  },

...
```

And this query result:

```sql
SELECT * FROM product WHERE category = 'homeware';

| SKU | prod_name |

| --- | --------- |

| 123 | iPad      |

| 456 | Ray-Bans  |
```

The enrichment would add this entity to your event:

```json
{

  "schema": "iglu:com.acme/products/jsonschema/1-0-0",

  "data": [

    {

      "SKU": "123",

      "prod_name": "iPad"

    },

    {

      "SKU": "456",

      "prod_name": "Ray-Bans"

    }

  ]

}
```

If you change the configuration to `EVERY_ROW`:

```json
...

  "output": {

    "expectedRows": "AT_LEAST_ZERO",

    "json": {

      "schema": "iglu:com.acme/product/jsonschema/1-0-0",

      "describes": "EVERY_ROW",

      "propertyNames": "LOWER_CASE"

    }

  }

...
```

Then two entities would be added to your event, one per row:

```json
{

  "schema": "iglu:com.acme/product/jsonschema/1-0-0",

  "data": {

    "sku": "123",

    "prod_name": "iPad"

  }

},

{

  "schema": "iglu:com.acme/product/jsonschema/1-0-0",

  "data": {

    "sku": "456",

    "prod_name": "Ray-Bans"

  }

}
```

---

# Event fingerprint enrichment
> Generate unique fingerprints for events using hash functions to enable deduplication and event matching.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/

This enrichment computes the fingerprint of an event using the Snowplow tracker payload.

Both the key and the value of all parameters in the payload are used to compute the fingerprint, except for the fields specified in `excludeParameters`. Check out the [atomic fields reference](/docs/fundamentals/canonical-event/) and [example HTTP requests](/docs/events/http-requests/) for more information on the relevant parameters.

This can be helpful when de-duplicating events.

## Configuration

The enrichment takes these parameters:

| Parameter           | Required | Description                                             |
| ------------------- | -------- | ------------------------------------------------------- |
| `hashAlgorithm`     | ✅        | The algorithm used to compute the event fingerprint.    |
| `excludeParameters` | ✅        | Parameters to exclude from the fingerprint calculation. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "excludeParameters": ["eid", "nuid", "stm", "cv"],

  "hashAlgorithm": "MD5"

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-0-1",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow",

    "name": "event_fingerprint_config",

    "enabled": true,

    "parameters": {

      "excludeParameters": ["eid", "nuid", "stm", "cv"],

      "hashAlgorithm": "MD5"

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

Supported hashing algorithms are:

- [MD5](https://en.wikipedia.org/wiki/MD5)
- [SHA1](https://en.wikipedia.org/wiki/SHA-1)
- [SHA256](https://en.wikipedia.org/wiki/SHA-2)
- [SHA384](https://en.wikipedia.org/wiki/SHA-2)
- [SHA512](https://en.wikipedia.org/wiki/SHA-2)

We recommend excluding the sent timestamp `stm`. When the tracker doesn't receive an acknowledgement after sending an event once, it will retry. The two copies of the event will have different `stm`s, but they're the same event.

Similarly, not much is gained by including the event ID, `eid`, in the hash given that this field is already used for de-duplication.

## Output

This enrichment will populate the `event_fingerprint` atomic event field.

---

# HTTP header extractor enrichment
> Extract HTTP request headers using regex patterns and attach them to events as entities.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/http-header-extractor-enrichment/

This enrichment can extract name/value pairs from the HTTP headers in the tracker request to the Collector, and attach them to the event as entities.

## Configuration

The enrichment takes one parameter:

| Parameter        | Required | Description                   |
| ---------------- | -------- | ----------------------------- |
| `headersPattern` | ✅        | A regex for matching headers. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "headersPattern": ".*"

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/http_header_extractor_config/jsonschema/1-0-0",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "name": "http_header_extractor_config",

    "enabled": false,

    "parameters": {

      "headersPattern": ".*"

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

All the headers that will match the regular expression defined in `headersPattern` parameter will be attached to the event.

In the example configuration, all the headers of the request would be attached to the event because of the `.*` regular expression pattern.

If you're only interested in the host, for example, use:

```json
      "headersPattern": "(?i)Host"
```

## Output

For **each** header matching the regex defined in `headersPattern`, an `http_header` entity is added to the enriched event. Each entity contains only one name/value pair.

### `http_header`

**Type:** Entity

Schema for a single HTTP header

**Properties:**

| Property         | Description |
| ---------------- | ----------- |
| `name` _string_  | _Required._ |
| `value` _string_ | _Required._ |

---

# IAB bot detection enrichment
> Identify bot and spider traffic using the IAB/ABC International Spiders and Bots List based on IP and user agent.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/iab-enrichment/

The IAB Spiders and Robots enrichment uses the [IAB/ABC International Spiders and Bots List](https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/) to determine whether an event was produced by a user or a robot/spider based on its IP address and user agent.

The Interactive Advertising Bureau (IAB) is an advertising business organization that develops industry standards, conducts research, and provides legal support for the online advertising industry.

Their internationally recognized list of spiders and bots is regularly maintained to try and identify the IP addresses of known bots and spiders.

## How the enrichment works

This enrichment performs several checks using the IAB database files and your custom override lists.

Here is the logic it uses. The values in parentheses are for the `reason` field in the IAB [entity](/docs/fundamentals/entities/) attached to the event.

```mermaid
flowchart TD
  IncList{{User agent in the custom **include** list?}}:::nowrap
  IncList -->|Yes| NotBot1(["Not bot"])
  IncList -->|No| ExcList{{User agent in the custom **exclude** list?}}:::nowrap
  ExcList -->|Yes| Bot1("**Bot**<br/>(FAILED_UA_EXCLUDE)")
  ExcList -->|No| IpFile{{IP address in the IAB IP file?}}:::nowrap
  IpFile -->|Yes| Bot2("**Bot**<br/>(FAILED_IP_EXCLUDE)")
  IpFile -->|No| NullUA{{User agent is null?}}:::nowrap
  NullUA -->|Yes| NotBot2(["Not bot"])
  NullUA -->|No| IncFile{{User agent in the IAB **include** file?}}:::nowrap
  IncFile -->|Yes| ExcFile{{User agent in the IAB **exclude** file?}}:::nowrap
  IncFile -->|No| Bot3("**Bot**<br/>(FAILED_UA_INCLUDE)")
  ExcFile -->|Yes| Bot4("**Bot**<br/>(FAILED_UA_EXCLUDE)")
  ExcFile -->|No| NotBot3(["Not bot"])
  classDef nowrap white-space:nowrap
```

A user agent string will match one of the lists or files if it contains a string from that list or file. The matching is case-insensitive.

For example, the user agent string `Chrome Chrome MyBot Chrome` will match an entry named `mybot`.

## Configuration

The enrichment takes these parameters:

| Parameter              | Required | Description                                                                                                                       |
| ---------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `ipFile`               | ✅        | Path to IP address exclude file. Already provided for CDI customers.                                                              |
| `excludeUseragentFile` | ✅        | Path to user agent exclude file. Already provided for CDI customers.                                                              |
| `includeUseragentFile` | ✅        | Path to user agent include file. Already provided for CDI customers.                                                              |
| `includeUseragents`    | ❌        | Additional user agent patterns to classify as browsers, extending `includeUseragentFile`. Case-insensitive substring match.       |
| `excludeUseragents`    | ❌        | Additional user agent patterns to classify as spiders/robots, extending `excludeUseragentFile`. Case-insensitive substring match. |

**Console:**

Configure the parameters in the Console enrichment editor. Keep the Console defaults for the `uri` fields. For example:

```json
{

  "ipFile": {

    "database": "ip_exclude_current_cidr.txt",

    "uri": "<use default value from Console>"

  },

  "excludeUseragentFile": {

    "database": "exclude_current.txt",

    "uri": "<use default value from Console>"

  },

  "includeUseragentFile": {

    "database": "include_current.txt",

    "uri": "<use default value from Console>"

  },

  "excludeUseragents": [

    "BotNotCaughtByIAB"

  ],

  "includeUseragents": [

    "MyServerSideTrackerThatIsNotABot"

  ]

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/iab_spiders_and_robots_enrichment/jsonschema/1-0-1",

  "data": {

    "name": "iab_spiders_and_robots_enrichment",

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "enabled": false,

    "parameters": {

      "ipFile": {

        "database": "ip_exclude_current_cidr.txt",

        "uri": "<your file location>"

      },

      "excludeUseragentFile": {

        "database": "exclude_current.txt",

        "uri": "<your file location>"

      },

      "includeUseragentFile": {

        "database": "include_current.txt",

        "uri": "<your file location>"

      },

      "excludeUseragents": [

        "BotNotCaughtByIAB"

      ],

      "includeUseragents": [

        "MyServerSideTrackerThatIsNotABot"

      ]

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### IAB files

> **Tip:** If you're using Snowplow CDI, you don't need to configure these. Use the default values provided in Console.

There are three configuration fields that correspond to the IAB/ABC database files:

| Field name             | Description                                                       |
| ---------------------- | ----------------------------------------------------------------- |
| `ipFile`               | Denylist of IP addresses considered to be robots or spiders       |
| `excludeUseragentFile` | Denylist of user agent strings considered to be robots or spiders |
| `includeUseragentFile` | Allowlist of user agent strings considered to be browsers         |

All three are mandatory and must have two inner fields:

- The `database` field containing the name of the database file.
- The `uri` field containing the URI of the bucket in which the database file is found. This field supports `http`, `https`, `gs`, and `s3` schemes.

The database filenames must be as follows:

| Field name                      | Filename                        |
| ------------------------------- | ------------------------------- |
| `ipFile.database`               | `"ip_exclude_current_cidr.txt"` |
| `excludeUseragentFile.database` | `"exclude_current.txt"`         |
| `includeUseragentFile.database` | `"include_current.txt"`         |

### Custom user agent lists

> **Note:** This feature is available since version 6.8.0 of Enrich.

In addition to the IAB database files, you can provide custom lists of user agent strings to supplement or override the detection. Two optional fields can be added to the `parameters` section:

| Field name          | Description                                                     |
| ------------------- | --------------------------------------------------------------- |
| `excludeUseragents` | A list of user agent strings to be treated as robots or spiders |
| `includeUseragents` | A list of user agent strings to be treated as browsers          |

Both fields accept a JSON array of strings. They are optional and default to empty lists if omitted.

A user agent matching `excludeUseragents` produces the following output values:

| Field           | Value               |
| --------------- | ------------------- |
| `spiderOrRobot` | `true`              |
| `category`      | `SPIDER_OR_ROBOT`   |
| `reason`        | `FAILED_UA_EXCLUDE` |
| `primaryImpact` | `UNKNOWN`           |

A user agent matching `includeUseragents` produces the following output values:

| Field           | Value        |
| --------------- | ------------ |
| `spiderOrRobot` | `false`      |
| `category`      | `BROWSER`    |
| `reason`        | `PASSED_ALL` |
| `primaryImpact` | `NONE`       |

Example:

```json
"excludeUseragents": ["my-custom-bot/1.0", "internal-crawler"],

"includeUseragents": ["my-legitimate-app/2.0"]
```

This is useful when you need to flag or allowlist specific user agents without modifying the IAB database files themselves.

## Input

This enrichment uses the following [fields](/docs/fundamentals/canonical-event/) of a Snowplow event:

- `useragent` to determine an event's user agent, which will be validated against the databases described in `excludeUseragentFile` and `includeUseragentFile`, as well as the custom lists in `excludeUseragents` and `includeUseragents`
- `user_ipaddress` to determine an event's IP address, which will be validated against the database described in `ipFile`
- `derived_tstamp` to determine an event's datetime. Some entries in the Spiders and Robots List can be considered "stale", and will be given a `category` of `INACTIVE_SPIDER_OR_ROBOT` rather than `ACTIVE_SPIDER_OR_ROBOT` based on their age

## Output

This enrichment adds a `spiders_and_robots` entity to the enriched event.

### `spiders_and_robots`

**Type:** Entity

Schema for an entity generated by the IAB Spiders and Robots enrichment

**Properties:**

| Property                  | Description                                                                                                                                                                                                                  |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spiderOrRobot` _boolean_ | _Required._ true if the IP address or user agent checked against the list is a spider or robot, false otherwise                                                                                                              |
| `category` __             | _Required._ Category based on activity if the IP/UA is a spider or robot, BROWSER otherwise Must be one of: `SPIDER_OR_ROBOT`, `ACTIVE_SPIDER_OR_ROBOT`, `INACTIVE_SPIDER_OR_ROBOT`, `BROWSER`                               |
| `reason` __               | _Required._ Type of failed check if the IP/UA is a spider or robot, PASSED\_ALL otherwise Must be one of: `FAILED_IP_EXCLUDE`, `FAILED_UA_INCLUDE`, `FAILED_UA_EXCLUDE`, `PASSED_ALL`                                        |
| `primaryImpact` __        | _Required._ Whether the spider or robot would affect page impression measurement, ad impression measurement, both or none Must be one of: `PAGE_IMPRESSIONS`, `AD_IMPRESSIONS`, `PAGE_AND_AD_IMPRESSIONS`, `UNKNOWN`, `NONE` |

---

# Available enrichments
> Browse the complete catalog of Snowplow enrichments that add entities and enhance event data during processing.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/

Snowplow offers a large number of enrichments that can be used to enhance your event data. Most enrichments attach data to the event as [entities](/docs/fundamentals/entities/), although some enrichments fill or modify event fields directly.

The order of enrichments is hard-coded and can't be changed. The table below lists the available enrichments **in the order they're executed**.

> **Note:** You can only configure one instance of each enrichment.

| Enrichment                                                                                                       | Description                                                                                                                              |
| ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/)                                          | Detect robots or spiders via the IAB/ABC International Spiders and Bots List.                                                            |
| User Agent utils                                                                                                 | This enrichment is deprecated. Please consider switching to [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/). |
| [UA Parser (legacy)](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)                     | Not recommended. Parse the user agent string and attach an entity with detailed user agent information. YAUAA enrichment is preferred.   |
| [Currency Conversion (legacy)](/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/) | Not recommended. Convert the values of legacy transaction events to a specified base currency using Open Exchange Rates API.             |
| [Referrer Parser](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)                  | Extract attribution data from referrer URLs.                                                                                             |
| [Campaign Attribution](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)        | Choose how query string parameters map to marketing campaign fields, e.g. `mkt_medium`.                                                  |
| [Cross Navigation](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/)                | Parse the extended cross navigation format in `_sp` query string parameter and attach the `cross_navigation` entity to an event.         |
| [Event Fingerprint](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)              | Generate a fingerprint for the event by hashing a set of fields. Helpful for deduplicating events.                                       |
| [Cookie Extractor](/docs/pipeline/enrichments/available-enrichments/cookie-extractor-enrichment/)                | Extract values of specific cookies into extra entities.                                                                                  |
| [HTTP Header Extractor](/docs/pipeline/enrichments/available-enrichments/http-header-extractor-enrichment/)      | Extract values of specific HTTP headers into extra entities.                                                                             |
| [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)                                      | Parse the user agent string and attach an entity with detailed user agent information.                                                   |
| [IP Lookup](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)                              | Look up useful data on the IP address in the MaxMind database.                                                                           |
| [ASN Lookup](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/)                            | Flag bot traffic by checking ASNs against known bad ASN lists.                                                                           |
| [Bot Detection](/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/)                      | Consolidate bot signals from YAUAA, IAB, and ASN lookup into a single entity.                                                            |
| [JavaScript](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/)                     | Write a JavaScript function which is executed for each event.                                                                            |
| [SQL Query](/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/)                             | Query an external relational database and attach the response as an extra entity.                                                        |
| [API Request](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/)                   | Query an external HTTP(S) API and attach the response as an extra entity.                                                                |
| [IP Anonymization](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/)                | Anonymize the IP addresses found in the `user_ipaddress` field by replacing some octets with `x`.                                        |
| [PII Pseudonymization](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)        | Pseudoanonymize specific fields to comply with data privacy regulation.                                                                  |

---

# IP anonymization enrichment
> Anonymize IP addresses by masking octets or segments to protect user privacy and comply with regulations.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/

This enrichment replaces the end of the user's IP address with "x"s, on a configurable length. The user IP is tracked in the `user_ipaddress` field of the atomic event.

For example, anonymizing one IPv4 octet would change an address of `255.255.255.255` to `255.255.255.x`, and anonymizing three octets would change it to `255.x.x.x`.

Both IPv4 and IPv6 are supported.

This enrichment runs after the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/).

## Configuration

The enrichment takes these parameters:

| Parameter      | Required | Description                           |
| -------------- | -------- | ------------------------------------- |
| `anonOctets`   | ✅        | Number of IPv4 octets to anonymize.   |
| `anonSegments` | ❌        | Number of IPv6 segments to anonymize. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "anonOctets": 1,

  "anonSegments": 1

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/anon_ip/jsonschema/1-0-1",

  "data": {

    "name": "anon_ip",

    "vendor": "com.snowplowanalytics.snowplow",

    "enabled": true,

    "parameters": {

      "anonOctets": 1,

      "anonSegments": 1

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Output

The anonymized value of the IP address is updated in-place in the `user_ipaddress` field, before ever being stored.

---

# IP lookup enrichment
> Resolve IP addresses to geographic locations and ISP information using MaxMind databases.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/

This enrichment uses [MaxMind](https://www.maxmind.com/en/geoip2-databases) databases to look up useful data based on the IP address collected by your Snowplow tracker.

MaxMind maintains databases of information like geographic location, second level domain names (e.g. `acme.com`), Internet Service Provider, organization name, and several other data points publicly associated with a given IP address.

Starting with Enrich 6.7.0, the enrichment supports ASN information, which is useful for detecting bot traffic coming from cloud computing providers. To enable this, provide either the ISP or the free ASN database through the `isp` or `asn` setting respectively.

## Select the MaxMind databases

MaxMind offers a free tier and a paid tier of databases.

From the free tier, you can provide two databases to Snowplow:

- [GeoLite2 City Database](https://dev.maxmind.com/geoip/docs/databases/city-and-country/), which contains geographic information (e.g. country) by IP address
- [GeoLite2 ASN Database](https://dev.maxmind.com/geoip/docs/databases/asn/), supported from Enrich 6.7.0, which contains autonomous system numbers by IP address

From the paid tier, you can provide four databases to Snowplow:

- [GeoIP2 City](https://dev.maxmind.com/geoip/docs/databases/city-and-country/), which also contains geographic information, but with more precision and coverage than the GeoLite2 City Database
- [GeoIP2 ISP](https://dev.maxmind.com/geoip/docs/databases/isp/), which contains information about the ISP serving that IP, and a more complete ASN mapping compared to the GeoLite2 ASN Database
- [GeoIP2 Domain](https://dev.maxmind.com/geoip/docs/databases/domain/), which contains information about the domain at that IP address
- [GeoIP2 Connection Type](https://dev.maxmind.com/geoip/docs/databases/connection-type/), which contains information about the connection type at that IP address.

## Host the databases in your cloud

> **Tip:** If you're a Snowplow CDI customer, the free-tier MaxMind database files are already provided and updated by Snowplow, so you don't need this step.
>
> There's a pre-configured URI for the directory containing these files. You can find it in the default enrichment configuration in [Console](https://console.snowplowanalytics.com).

Once downloaded, take the `.mmdb` files and upload them to a location on your cloud:

- Amazon S3 if running Snowplow on AWS e.g. `s3://my-private-bucket/third-party/maxmind`
- Azure ADLS if running Snowplow on Azure e.g. `https://my-private-storage-container.dfs.core.windows.net/third-party/maxmind`
- Google Cloud Storage if running Snowplow on GCS e.g. `gs://my-private-bucket/third-party/maxmind`

If the databases need updating in future, download the latest version and overwrite this file in your storage.

MaxMind also offer a method to [download and update their databases programmatically](https://dev.maxmind.com/geoip/updating-databases/#using-geoip-update).

## Configure the enrichment

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).
>
> Note that to test this enrichment, you will need events with realistic IP addresses, not local ones like `192.168.0.42`.
>
> This isn't an issue if running Micro through Console, but will be trickier if running locally.
>
> If you are using a web browser to test your site or app, you can spoof a specific IP address using a browser plugin that sets an `X-Forwarded-For` header. For example, here are plugins for [Chrome](https://chrome.google.com/webstore/detail/x-forwarded-for-header/hkghghbnihliadkabmlcmcgmffllglin/) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/x-forwarded-for-injector/). Install the plugin and set the IP address to your liking.
>
> Alternatively, you can [set up Micro to receive external IP addresses](/docs/testing/snowplow-micro/local/remote-usage/#exposing-micro-via-a-public-domain-name).

The enrichment takes these parameters:

| Parameter        | Required | Description                              |
| ---------------- | -------- | ---------------------------------------- |
| `geo`            | ❌        | MaxMind GeoIP2 / GeoLite2 City database. |
| `isp`            | ❌        | MaxMind GeoIP2 ISP database.             |
| `domain`         | ❌        | MaxMind GeoIP2 Domain database.          |
| `connectionType` | ❌        | MaxMind GeoIP2 Connection Type database. |
| `asn`            | ❌        | MaxMind GeoLite2 ASN database.           |

**Console:**

Configure the parameters in the Console enrichment editor. Keep the Console defaults for the `uri` fields, unless you are using a custom paid database. For example:

```json
{

  "geo": {

    "database": "GeoLite2-City.mmdb",

    "uri": "<use default value from Console>"

  },

  "asn": {

    "database": "GeoLite2-ASN.mmdb",

    "uri": "<use default value from Console>"

  }

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-1",

  "data": {

    "name": "ip_lookups",

    "vendor": "com.snowplowanalytics.snowplow",

    "enabled": true,

    "parameters": {

      "geo": {

        "database": "GeoLite2-City.mmdb",

        "uri": "<your file location>"

      },

      "asn": {

        "database": "GeoLite2-ASN.mmdb",

        "uri": "<your file location>"

      }

    }

  }

}
```

***

Each of the parameters is an object with two required fields: `database` and `uri`:

- The `database` field is an enum that contains the name of the MaxMind database file.
- The `uri` field contains the URI of the bucket in which the database file is found. This can have either `http:` or `s3:` or `gs:` as the scheme, and must not end with a trailing slash.

> **Tip:** If you are using Snowplow CDI and the free MaxMind databases, use the same default value you can see in Console for all `uri` fields, whether `geo`, `asn`, etc.
>
> Otherwise, provide the location (e.g. `s3://my-private-bucket/third-party/maxmind`) where you uploaded the files.

Allowed database filenames are as follows. If the filename you provide isn't one of these, the enrichment JSON will fail validation, causing a failed event.

| Parameter        | Valid database names                                  |
| ---------------- | ----------------------------------------------------- |
| `geo`            | `GeoLite2-City.mmdb` (free) `GeoIP2-City.mmdb` (paid) |
| `isp`            | `GeoIP2-ISP.mmdb`                                     |
| `domain`         | `GeoIP2-Domain.mmdb`                                  |
| `connectionType` | `GeoIP2-Connection-Type.mmdb`                         |
| `asn`            | `GeoLite2-ASN.mmdb`                                   |

## Output

This enrichment populates the [atomic table fields prefixed with `geo_` and `ip_`](/docs/fundamentals/canonical-event/#ip-address-fields).

If ASN data is available, the enrichment also adds an `asn` entity to the enriched event.

### `asn`

**Type:** Entity

Schema for ASN entity generated by IP lookup enrichment

**Properties:**

| Property                | Description                                                                                                                 |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `number` _integer_      | _Required._ The autonomous system number associated with the IP address                                                     |
| `organization` _string_ | _Optional._ The organization associated with the registered autonomous system number for the IP address                     |
| `likelyBot` _boolean_   | _Optional._ Populated by the ASN lookup enrichment. Set to true if the ASN belongs to hosting providers, data centers, etc. |

---

# PII pseudonymization enrichment
> Pseudonymize personally identifiable information in events using hashing for privacy compliance.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/

The PII (personally identifiable information) pseudonymization enrichment runs after all the other enrichments, and pseudonymizes the fields that are configured as PII.

It enables you to better protect the privacy rights of data subjects and comply with data regulation.

In Europe the obligations regarding Personal Data handling have been outlined on the [GDPR EU website](https://www.gdpreu.org/the-regulation/key-concepts/personal-data/).

> **Tip:** Hashing a field can change its format (e.g. email) and its length. This could make a whole valid original event invalid if its schema is not compatible with the hashing.

## Configuration

For historical reasons, the configuration uses terms that are no longer used elsewhere in Snowplow.

The enrichment takes these parameters:

| Parameter       | Required | Description                                                                                                                                         |
| --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pii`           | ✅        | List of all the fields for which pseudonymization will be performed.                                                                                |
| `strategy`      | ✅        | The pseudonymization strategy which will be applied to all the fields specified in the `pii` section.                                               |
| `anonymousOnly` | ❌        | Not configurable in Console: whether to enable anonymous mode. If enabled, the enrichment masks PII only when the `SP-Anonymous` header is present. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "pii": [

    {

      "pojo": {

        "field": "user_id"

      }

    },

    {

      "pojo": {

        "field": "user_ipaddress"

      }

    },

    {

      "json": {

        "field": "unstruct_event",

        "schemaCriterion": "iglu:com.google.analytics.measurement-protocol/user/jsonschema/1-*-*",

        "jsonPath": "$.['clientId', 'userId']"

      }

    }

  ],

  "strategy": {

    "pseudonymize": {

      "hashFunction": "SHA-1",

      "salt": "pepper123"

    }

  }

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/pii_enrichment_config/jsonschema/2-0-1",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "name": "pii_enrichment_config",

    "emitEvent": false,

    "enabled": true,

    "anonymousOnly": false,

    "parameters": {

      "pii": [

        {

          "pojo": {

            "field": "user_id"

          }

        },

        {

          "pojo": {

            "field": "user_ipaddress"

          }

        },

        {

          "json": {

            "field": "unstruct_event",

            "schemaCriterion": "iglu:com.google.analytics.measurement-protocol/user/jsonschema/1-*-*",

            "jsonPath": "$.['clientId', 'userId']"

          }

        }

      ],

      "strategy": {

        "pseudonymize": {

          "hashFunction": "SHA-1",

          "salt": "pepper123"

        }

      }

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

Note that the legacy `emitEvent` field is ignored by Enrich.

### `pii` fields

The enrichment can act on most properties in the event:

- A defined subset of [atomic event properties](/docs/fundamentals/canonical-event/), including `user_id`, see below
- [Self-describing event](/docs/fundamentals/events/#self-describing-events) fields
- [Entities](/docs/fundamentals/entities/) attached by tracker SDKs
- Entities attached by other enrichments

Specify `pojo` to hash atomic event fields. The available fields are:

- `user_id`
- `user_ipaddress`
- `user_fingerprint`
- `domain_userid`
- `network_userid`
- `refr_domain_userid`
- `domain_sessionid`
- `ip_organization`
- `ip_domain`
- `mkt_term`
- `mkt_content`
- `mkt_clickid`
- `se_category`
- `se_action`
- `se_label`
- `se_property`
- `tr_orderid`
- `ti_orderid`

Providing a field not in this list will cause a failed event.

For `json`, specify the field name as either `unstruct_event` for self-describing event fields, `contexts` for fields in entities added during tracking, or `derived_contexts` for fields in enrichment entities. Add two additional fields:

- `schemaCriterion` is the Iglu schema URI. You can specify all versions of the schema (`*-*-*`), or a specific major version (e.g. `1-*-*`), major plus minor (e.g. `1-1-*`) or a full major-minor-patch version (e.g. `1-1-1`)
- `jsonPath` is the [JSON Path statement](http://goessner.net/articles/JsonPath/) to navigate to the field or fields inside the JSON that you want to hash.

The resolved values should be primitive types (string, number, or boolean).

### `strategy`

Choose which pseudonymization hashing algorithm strategy to use:

- `MD5`: the 128-bit algorithm [MD5](https://en.wikipedia.org/wiki/MD5#MD5_hashes)
- `SHA-1:` the 160-bit algorithm [SHA-1](https://en.wikipedia.org/wiki/SHA-1#Example_hashes)
- `SHA-256`: 256-bit variant of the [SHA-2](https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions) algorithm
- `SHA-384`: 384-bit variant of the [SHA-2](https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions) algorithm
- `SHA-512`: 512-bit variant of the [SHA-2](https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions) algorithm
- `MD2`: the 128-bit algorithm [MD2](https://en.wikipedia.org/wiki/MD2_\(cryptography\)#MD2_hashes) (not recommended due to performance reasons: see [RFC6149](https://tools.ietf.org/html/rfc6149))

Once you've started using the enrichment, it's important to keep the same `salt` value in order to maintain the same hashed output for the same input. This is crucial for use cases where you need to join events on hashed PII fields, or simply to avoid having different hashed values for the same original value across events.

### `anonymousOnly` mode

> **Note:** This parameter isn't configurable in Console.

By default, `anonymousOnly` is `false`, i.e. PII fields are always masked.

When `anonymousOnly` is set to `true`, PII fields are masked only in events tracked in anonymous mode, i.e. when the `SP-Anonymous` header is present.

This is useful for compliance with regulations such as GDPR, where you'd start with [anonymous tracking](/docs/events/anonymous-tracking/) by default, and switch to non-anonymous tracking when the user consents to data collection.

This feature was added in Enrich 5.3.0.

## Output

The fields are updated in place in the event. The original values aren't stored.

---

# Referrer parser enrichment
> Extract attribution data from referrer URLs to identify traffic sources, search terms, and marketing channels.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/

This enrichment uses the [Snowplow referer-parser](https://github.com/snowplow/referer-parser) library to extract attribution data from referrer URLs.

This is particularly useful when looking for traffic from specific search engine providers or social networks.

## Configuration

The enrichment takes these parameters:

| Parameter         | Required | Description                                                                         |
| ----------------- | -------- | ----------------------------------------------------------------------------------- |
| `internalDomains` | ✅        | Subdomains to classify as `Internal` traffic sources.                               |
| `database`        | ✅        | Filename of the referer-parser database. Already provided for CDI customers.        |
| `uri`             | ✅        | URI of the bucket containing the database file. Already provided for CDI customers. |
| `referrers`       | ❌        | Custom referrer-to-category mappings, taking precedence over the database.          |

**Console:**

Configure the parameters in the Console enrichment editor. Keep the Console default for the `database` and `uri` fields. For example:

```json
{

  "internalDomains": [],

  "database": "<use default value from Console>",

  "uri": "<use default value from Console>",

  "referrers": {

    "search": {

      "Search website 1": {

        "domains": ["search.acme.com"],

        "parameters": ["q"]

      },

      "Search website 2": {

        "domains": ["search.acmebis.com"]

      }

    },

    "social": {

      "Social website": {

        "domains": ["social.acme.com"]

      }

    }

  }

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/referer_parser/jsonschema/2-0-1",

  "data": {

    "name": "referer_parser",

    "vendor": "com.snowplowanalytics.snowplow",

    "enabled": true,

    "parameters": {

      "internalDomains": [],

      "database": "referers-5.3.json",

      "uri": "https://s3-eu-west-1.amazonaws.com/snowplow-hosted-assets/third-party/referer-parser/",

      "referrers": {

        "search": {

          "Search website 1": {

            "domains": ["search.acme.com"],

            "parameters": ["q"]

          },

          "Search website 2": {

            "domains": ["search.acmebis.com"]

          }

        },

        "social": {

          "Social website": {

            "domains": ["social.acme.com"]

          }

        }

      }

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### `internalDomains`

Use this property to specify a list of subdomains to class as `Internal` traffic sources.

```json
"internalDomains": [

    "community.snowplow.io",

    "docs.snowplow.io"

],
```

> **Note:** The enrichment will also classify `refr_medium` as `Internal` when an event's `page_urlhost` matches its `refr_urlhost`, regardless of the configured `internalDomains`.
>
> This behavior isn't configurable, and may require handling in data models or a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/) to change.

### `database` and `uri`

> **Tip:** If you're using Snowplow CDI, you don't need to configure these. Use the default values provided in Console.

Provide details of the referer-parser format database to use. Snowplow hosts a database you can use: the latest version is listed in the [library README](https://github.com/snowplow/referer-parser). Alternatively, the enrichment will accept any valid JSON or YAML file in the right format.

### Custom referrer mappings

> **Note:** This feature is available since version 6.9.0 of Enrich.

You can add your own referrer-to-category mappings directly in the enrichment configuration using the optional `referrers` parameter. This is useful when you need to classify new traffic sources - such as internal tools, niche search engines, or AI chatbots - without waiting for changes to the upstream database.

Custom mappings take precedence over the default database. If a domain appears in both your custom mappings and the default database, the custom mapping is used.

The `referrers` parameter is a nested object structured like this:

```json
"referrers": {

  "<medium>": {

    "<source name>": {

      "domains": ["<domain1>", "<domain2>"],

      "parameters": ["<param1>"]

    }

  }

}
```

| Field           | Description                                                                                                       |
| --------------- | ----------------------------------------------------------------------------------------------------------------- |
| `<medium>`      | The referrer category e.g., `search`, `social`, `email`. This value populates `refr_medium`.                      |
| `<source name>` | A human-readable name for the source e.g., `"Google"`, `"Internal Search"`. This value populates `refr_source`.   |
| `domains`       | An array of hostnames to match against the referrer URL. At least one domain is required.                         |
| `parameters`    | An optional array of URL query parameter names to extract search terms from. Matched values populate `refr_term`. |

For example, to classify a custom search engine and a social network:

```json
"referrers": {

  "search": {

    "Corporate Search": {

      "domains": ["search.example.com"],

      "parameters": ["q", "query"]

    }

  },

  "social": {

    "Internal Forum": {

      "domains": ["forum.example.com"]

    }

  }

}
```

With this configuration, a referrer URL of `https://search.example.com/?q=snowplow` would produce the following:

| Field         | Value              |
| ------------- | ------------------ |
| `refr_medium` | `search`           |
| `refr_source` | `Corporate Search` |
| `refr_term`   | `snowplow`         |

> **Tip:** You can use custom referrer mappings to immediately test new categorizations in your pipeline. Once validated, consider contributing your mappings back to the [upstream referer-parser database](https://github.com/snowplow/referer-parser) via a pull request.

## Output

This enrichment populates the `refr_medium`, `refr_source`, and `refr_term` [atomic event fields](/docs/fundamentals/canonical-event/#page-fields).

---

# UA parser enrichment (legacy)
> Parse user agent strings to extract browser, operating system, and device information.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/

> **Tip:** The [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) supersedes this enrichment and is preferable in most cases.

This enrichment uses the [ua-parser library](https://github.com/ua-parser/uap-core/) to parse the user agent and provide information about the user's device.

Both the UA parser and YAUAA enrichments will add an additional entity to the event, based on the information available in the `User-Agent` header of the tracker request, or the `ua` [tracker payload field](/docs/fundamentals/canonical-event/#browser-fields). Unlike YAUAA, this enrichment doesn't support client hints.

## UA parser vs YAUAA

This table compares the output of the two user agent parsing enrichments. The UA parser enrichment provides fewer fields, but is more lightweight and has lower memory requirements than YAUAA.

The example output is based on the user agent string `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0` (Firefox 106 on Mac OS X).

| UA Parser field    | YAUAA field                       | Example         |
| ------------------ | --------------------------------- | --------------- |
| `deviceFamily`     | `deviceClass`                     | Desktop         |
| ❌                  | `deviceName`                      | Apple Macintosh |
| ❌                  | `deviceBrand`                     | Apple           |
| ❌                  | `deviceCpu`                       | Intel           |
| ❌                  | `deviceCpuBits`                   | 32              |
| ❌                  | `deviceFirmwareVersion`           |                 |
| ❌                  | `deviceVersion`                   |                 |
| ❌                  | `operatingSystemClass`            | Desktop         |
| `osFamily`         | `operatingSystemName`             | Mac OS X        |
| `osVersion`        | `operatingSystemVersion`          | 10.15           |
| `osMajor`          | `operatingSystemVersionMajor`     | 10              |
| `osMinor`          | ❌                                 | 15              |
| `osPatch`          | ❌                                 |                 |
| `osPatchMinor`     | `operatingSystemVersionBuild`     |                 |
| ❌                  | `operatingSystemNameVersion`      | Mac OS X 10.15  |
| ❌                  | `operatingSystemNameVersionMajor` | Mac OS X 10     |
| ❌                  | `layoutEngineClass`               | Browser         |
| ❌                  | `layoutEngineName`                | Gecko           |
| ❌                  | `layoutEngineVersion`             | 106.0           |
| ❌                  | `layoutEngineVersionMajor`        | 106             |
| ❌                  | `layoutEngineNameVersion`         | Gecko 106.0     |
| ❌                  | `layoutEngineNameVersionMajor`    | Gecko 106       |
| ❌                  | `layoutEngineBuild`               | 20100101        |
| ❌                  | `agentClass`                      | Browser         |
| `useragentFamily`  | `agentName`                       | Firefox         |
| `useragentVersion` | `agentVersion`                    | 106.0           |
| `useragentMajor`   | `agentVersionMajor`               | 106             |
| `useragentMinor`   | ❌                                 | 0               |
| `useragentPatch`   | ❌                                 |                 |
| ❌                  | `agentNameVersion`                | Firefox 106.0   |
| ❌                  | `agentNameVersionMajor`           | Firefox 106     |
| ❌                  | `agentBuild`                      |                 |
| ❌                  | `agentLanguage`                   |                 |
| ❌                  | `agentLanguageCode`               |                 |
| ❌                  | `agentInformationEmail`           |                 |
| ❌                  | `agentInformationUrl`             |                 |
| ❌                  | `agentSecurity`                   |                 |
| ❌                  | `agentUuid`                       |                 |
| ❌                  | `webviewAppName`                  |                 |
| ❌                  | `webviewAppVersion`               |                 |
| ❌                  | `webviewAppVersionMajor`          |                 |
| ❌                  | `webviewAppNameVersionMajor`      |                 |
| ❌                  | `facebookCarrier`                 |                 |
| ❌                  | `facebookDeviceClass`             |                 |
| ❌                  | `facebookDeviceName`              |                 |
| ❌                  | `facebookDeviceVersion`           |                 |
| ❌                  | `facebookFBOP`                    |                 |
| ❌                  | `facebookFBSS`                    |                 |
| ❌                  | `facebookOperatingSystemName`     |                 |
| ❌                  | `facebookOperatingSystemVersion`  |                 |
| ❌                  | `anonymized`                      |                 |
| ❌                  | `hackerAttackVector`              |                 |
| ❌                  | `hackerToolkit`                   |                 |
| ❌                  | `koboAffiliate`                   |                 |
| ❌                  | `koboPlatformId`                  |                 |
| ❌                  | `iECompatibilityVersion`          |                 |
| ❌                  | `iECompatibilityVersionMajor`     |                 |
| ❌                  | `iECompatibilityNameVersion`      |                 |
| ❌                  | `iECompatibilityNameVersionMajor` |                 |
| ❌                  | `carrier`                         |                 |
| ❌                  | `gSAInstallationID`               |                 |
| ❌                  | `networkType`                     |                 |

## Configuration

The optional `database` and `uri` parameters allow you to specify a custom ruleset. By default, the enrichment will use the one defined in the [ua-parser](https://github.com/ua-parser/uap-core/) library.

The enrichment takes these parameters:

| Parameter  | Required | Description                                           |
| ---------- | -------- | ----------------------------------------------------- |
| `database` | ❌        | Filename of a custom ruleset.                         |
| `uri`      | ❌        | URI of the bucket containing the custom ruleset file. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "database": "regexes-latest.yaml",

  "uri": "s3://snowplow-hosted-assets/third-party/ua-parser/"

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/ua_parser_config/jsonschema/1-0-1",

  "data": {

    "vendor": "com.snowplowanalytics.snowplow",

    "name": "ua_parser_config",

    "enabled": true,

    "parameters": {

      "database": "regexes-latest.yaml",

      "uri": "s3://snowplow-hosted-assets/third-party/ua-parser/"

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Output

This enrichment adds a `ua_parser_context` entity to the event. The example data shows the output for a `User-Agent` string of

```txt
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
```

### `ua_parser_context`

**Type:** Entity

Schema for useragent context generated by ua-parser enrichment

**Properties:**

| Property                    | Description                                      |
| --------------------------- | ------------------------------------------------ |
| `useragentFamily` _string_  | _Required._ Useragent family (browser) name      |
| `useragentMajor` _string_   | _Required._ Useragent major version              |
| `useragentMinor` _string_   | _Required._ Useragent minor version              |
| `useragentPatch` _string_   | _Optional._ Useragent patch version              |
| `useragentVersion` _string_ | _Optional._ Full version of the useragent        |
| `osFamily` _string_         | _Required._ Operating system name                |
| `osMajor` _string_          | _Optional._ Operating system major version       |
| `osMinor` _string_          | _Optional._ Operating system minor version       |
| `osPatch` _string_          | _Optional._ Operating system patch version       |
| `osPatchMinor` _string_     | _Optional._ Operating system patch minor version |
| `osVersion` _string_        | _Optional._ Operating system full version        |
| `deviceFamily` _string_     | _Required._ Device type                          |

---

# YAUAA enrichment
> Parse user agent strings with advanced detection using Yet Another UserAgent Analyzer for detailed device insights.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/

The YAUAA (Yet Another User Agent Analyzer) enrichment is a user agent parser and analyzer. It uses the [YAUAA library](https://yauaa.basjes.nl/). The library is stored in memory; there is no interaction with an external system.

If you don't need the full YAUAA output, consider using the [UA parser enrichment](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/) instead.

This enrichment adds an additional entity to the event.

## Input

This enrichment uses the following inputs:

- The `User-Agent` HTTP header of the tracker request.
- The `useragent` field from the [tracker payload](/docs/events/ootb-data/device-and-browser/), if set. This field has priority over the `User-Agent` HTTP header.
- [Client hint](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Client_hints) HTTP request headers.

The supported client hints headers are:

- `Sec-CH-UA` (low entropy)
- `Sec-CH-UA-Mobile` (low entropy)
- `Sec-CH-UA-Platform` (low entropy)
- `Sec-CH-UA-Full-Version-List`
- `Sec-CH-UA-Full-Version` (deprecated)
- `Sec-CH-UA-Arch`
- `Sec-CH-UA-Bitness`
- `Sec-CH-UA-Model`
- `Sec-CH-UA-Platform-Version`

## Configure high-entropy client hints

The YAUAA enrichment does not require you to make any changes to your website, if you are happy with the level of detail you get from the default user agent strings. However, it is possible to improve the accuracy of the user agent data produced by YAUAA, if you are able to configure your website to send [high-entropy client hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Client_hints#high_entropy_hints) to the Collector by following these instructions.

Chromium browsers will include low-entropy client hint HTTP request headers by default. To include the more detailed high-entropy client hint headers, additional configuration is required: specifying the hints to include with `Permissions-Policy` and `Accept-CH`.

Update the URLs to your Collector, and add this configuration to your web server:

```txt
Accept-CH: sec-ch-ua-full-version-list, sec-ch-ua-platform-version, sec-ch-ua-arch, sec-ch-ua-bitness, sec-ch-ua-model

Permissions-Policy: ch-ua-full-version-list=("https://<YOUR COLLECTOR>"), ch-ua-platform-version=("https://<YOUR COLLECTOR>"), ch-ua-arch=("https://<YOUR COLLECTOR>"), ch-ua-bitness=("https://<YOUR COLLECTOR>"), ch-ua-model=("https://<YOUR COLLECTOR>")
```

Alternatively, add the following `meta` tag with `Delegate-CH` to the header section of your site's HTML, updating the URLs to your Collector:

```html
<meta http-equiv="Delegate-CH" content="sec-ch-ua-full-version-list https://<YOUR COLLECTOR>; sec-ch-ua-platform-version https://<YOUR COLLECTOR>; sec-ch-ua-arch https://<YOUR COLLECTOR>; sec-ch-ua-bitness https://<YOUR COLLECTOR>; sec-ch-ua-model https://<YOUR COLLECTOR>">
```

## Configuration

The enrichment takes one parameter:

| Parameter   | Required | Description                                                                                                                  |
| ----------- | -------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `cacheSize` | ❌        | Number of already parsed user agents to keep in memory for faster processing. Default is 10000. Set to 0 to disable caching. |

**Console:**

Configure the parameters in the Console enrichment editor. For example:

```json
{

  "cacheSize": 10000

}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/yauaa_enrichment_config/jsonschema/1-0-0",

  "data": {

    "enabled": true,

    "vendor": "com.snowplowanalytics.snowplow.enrichments",

    "name": "yauaa_enrichment_config",

    "parameters": {

      "cacheSize": 10000

    }

  }

}
```

***

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Output

This enrichment adds a `yauaa_context` entity to the event.

If a field can't be determined, it won't be included. However, some fields can have the value `UNKNOWN`. The only field that will always be present is `deviceClass`. Check out the [YAUAA documentation](https://yauaa.basjes.nl/expect/fieldvalues/) for more details on the output fields.

### `yauaa_context`

**Type:** Entity

Schema for an entity generated by the YAUAA enrichment after parsing the user agent

**Properties:**

| Property                                   | Description                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deviceClass` __                           | _Required._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Desktop`, `Anonymized`, `Unknown`, `UNKNOWN`, `Mobile`, `Tablet`, `Phone`, `Watch`, `Virtual Reality`, `eReader`, `Set-top box`, `TV`, `Game Console`, `Handheld Game Console`, `Voice`, `Robot`, `Robot Mobile`, `Spy`, `Hacker` |
| `deviceName` _string_                      | _Optional._ Example: Google Nexus 6                                                                                                                                                                                                                                                                            |
| `deviceBrand` _string_                     | _Optional._ Example: Google                                                                                                                                                                                                                                                                                    |
| `deviceCpu` _string_                       | _Optional._                                                                                                                                                                                                                                                                                                    |
| `deviceCpuBits` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                    |
| `deviceFirmwareVersion` _string_           | _Optional._                                                                                                                                                                                                                                                                                                    |
| `deviceVersion` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                    |
| `operatingSystemClass` __                  | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Desktop`, `Mobile`, `Cloud`, `Embedded`, `Game Console`, `Hacker`, `Anonymized`, `Unknown`                                                                                                                                        |
| `operatingSystemName` _string_             | _Optional._ Examples: Linux, Android.                                                                                                                                                                                                                                                                          |
| `operatingSystemVersion` _string_          | _Optional._                                                                                                                                                                                                                                                                                                    |
| `operatingSystemNameVersion` _string_      | _Optional._                                                                                                                                                                                                                                                                                                    |
| `operatingSystemVersionBuild` _string_     | _Optional._                                                                                                                                                                                                                                                                                                    |
| `layoutEngineClass` __                     | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Browser`, `Mobile App`, `Hacker`, `Robot`, `Unknown`                                                                                                                                                                              |
| `layoutEngineName` _string_                | _Optional._                                                                                                                                                                                                                                                                                                    |
| `layoutEngineVersion` _string_             | _Optional._                                                                                                                                                                                                                                                                                                    |
| `layoutEngineVersionMajor` _string_        | _Optional._                                                                                                                                                                                                                                                                                                    |
| `layoutEngineNameVersion` _string_         | _Optional._                                                                                                                                                                                                                                                                                                    |
| `layoutEngineNameVersionMajor` _string_    | _Optional._                                                                                                                                                                                                                                                                                                    |
| `layoutEngineBuild` _string_               | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentClass` __                            | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Browser`, `Browser Webview`, `Mobile App`, `Robot`, `Robot Mobile`, `Cloud Application`, `Email Client`, `Voice`, `Special`, `Testclient`, `Hacker`, `Unknown`                                                                    |
| `agentName` _string_                       | _Optional._ Example: Chrome.                                                                                                                                                                                                                                                                                   |
| `agentVersion` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentVersionMajor` _string_               | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentNameVersion` _string_                | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentNameVersionMajor` _string_           | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentBuild` _string_                      | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentLanguage` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentLanguageCode` _string_               | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentInformationEmail` _string_           | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentInformationUrl` _string_             | _Optional._                                                                                                                                                                                                                                                                                                    |
| `agentSecurity` _string_                   | _Optional._ Must be one of: `Weak security`, `Strong security`, `Unknown`, `Hacker`                                                                                                                                                                                                                            |
| `agentUuid` _string_                       | _Optional._                                                                                                                                                                                                                                                                                                    |
| `webviewAppName` _string_                  | _Optional._                                                                                                                                                                                                                                                                                                    |
| `webviewAppVersion` _string_               | _Optional._                                                                                                                                                                                                                                                                                                    |
| `webviewAppVersionMajor` _string_          | _Optional._                                                                                                                                                                                                                                                                                                    |
| `webviewAppNameVersionMajor` _string_      | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookCarrier` _string_                 | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceClass` _string_             | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceName` _string_              | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceVersion` _string_           | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookFBOP` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookFBSS` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookOperatingSystemName` _string_     | _Optional._                                                                                                                                                                                                                                                                                                    |
| `facebookOperatingSystemVersion` _string_  | _Optional._                                                                                                                                                                                                                                                                                                    |
| `anonymized` _string_                      | _Optional._                                                                                                                                                                                                                                                                                                    |
| `hackerAttackVector` _string_              | _Optional._                                                                                                                                                                                                                                                                                                    |
| `hackerToolkit` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                    |
| `koboAffiliate` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                    |
| `koboPlatformId` _string_                  | _Optional._                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityVersion` _string_          | _Optional._                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityVersionMajor` _string_     | _Optional._                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityNameVersion` _string_      | _Optional._                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityNameVersionMajor` _string_ | _Optional._                                                                                                                                                                                                                                                                                                    |
| `carrier` _string_                         | _Optional._                                                                                                                                                                                                                                                                                                    |
| `gSAInstallationID` _string_               | _Optional._                                                                                                                                                                                                                                                                                                    |
| `networkType` _string_                     | _Optional._                                                                                                                                                                                                                                                                                                    |
| `operatingSystemNameVersionMajor` _string_ | _Optional._                                                                                                                                                                                                                                                                                                    |
| `operatingSystemVersionMajor` _string_     | _Optional._                                                                                                                                                                                                                                                                                                    |

---

# Introduction to enrichments
> Configure enrichments to add extra data to your Snowplow events.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/

During enrichment, events are validated against their schemas, and additional data is attached to the events.

Snowplow runs two kinds of enrichments:

- **Configurable enrichments**, which you can opt into. Most produce entities attached to the event, though a few modify atomic event fields directly.
- **Hardcoded enrichments**, which always run as part of the common enrichment process. They populate [atomic event fields](/docs/fundamentals/canonical-event/).

## Configurable enrichments

See the [available enrichments](/docs/pipeline/enrichments/available-enrichments/) page for the full list, in execution order.

## Hardcoded enrichment fields

During enrichment, the [Enrich](/docs/api-reference/enrichment-components/) application adds a number of fields to the event.

### Always populated

The following fields are always populated during the enrichment process:

| Added parameter    | Purpose                                            |
| ------------------ | -------------------------------------------------- |
| `v_etl`            | Version of the enrichment process.                 |
| `etl_tstamp`       | Timestamp the event began enrichment.              |
| `collector_tstamp` | Timestamp for the event recorded by the Collector. |
| `derived_tstamp`   | Calculated timestamp.                              |

See the [Timestamps overview page](/docs/events/timestamps/) for more details on Snowplow timestamps.

### Conditionally populated

Some fields are populated depending on whether the tracker provided the corresponding value or not.

These parameters can be provided by the tracker or by the [Collector](/docs/pipeline/collector/).

| Raw parameter | Enriched parameter | Purpose                                                                                                                               |
| ------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `eid`         | `event_id`         | The unique event identifier (UUID). Trackers should always assign this value, but it can be assigned during enrichment as a fallback. |
| `cv`          | `v_collector`      | Collector type/version.                                                                                                               |
| `tnuid`       | `network_userid`   | User ID set by Snowplow using a third-party cookie. Overwritten with tracker-set `tnuid`.                                             |
| `ip`          | `user_ipaddress`   | The Collector logs IP address by default. You can override the value derived from the Collector by populating this in the tracker.    |
| `ua`          | `useragent`        | Raw user agent (browser string). Can be overwritten with `ua`.                                                                        |

The raw tracker payload parameters `res`, `ds`, and `vp` each combine a width and height into a single string, e.g. `1280x1024`. During enrichment, each is split into two separate integer fields.

| Raw parameter | Added parameters                        | Purpose                 |
| ------------- | --------------------------------------- | ----------------------- |
| `res`         | `dvce_screenwidth`, `dvce_screenheight` | Screen or monitor size. |
| `ds`          | `doc_width`, `doc_height`               | Page size.              |
| `vp`          | `br_viewwidth`, `br_viewheight`         | Viewport size.          |

The `url` parameter provides the [atomic `page_url` field](/docs/fundamentals/canonical-event/#page-fields). During enrichment, the following parts are extracted to populate separate fields:

| Added parameter    | Purpose                                          |
| ------------------ | ------------------------------------------------ |
| `page_urlscheme`   | Scheme (protocol), for example `https`.          |
| `page_urlhost`     | Host (domain), for example `www.snowplow.io`.    |
| `page_urlport`     | Port if specified, 443 or 80 if not.             |
| `page_urlpath`     | Path to page, for example `/product/index.html`. |
| `page_urlquery`    | Query string, for example `id=GTM-DLRG`.         |
| `page_urlfragment` | Fragment (anchor), for example `4-conclusion`.   |

Similarly, the `refr` parameter provides the [atomic `page_referrer` field](/docs/fundamentals/canonical-event/#page-fields). During enrichment, the following parts are extracted to populate separate fields:

| Added parameter    | Purpose                              |
| ------------------ | ------------------------------------ |
| `refr_urlscheme`   | Scheme (protocol).                   |
| `refr_urlhost`     | Host (domain).                       |
| `refr_urlport`     | Port if specified, 443 or 80 if not. |
| `refr_urlpath`     | Path to page.                        |
| `refr_urlquery`    | Query string.                        |
| `refr_urlfragment` | Fragment (anchor).                   |

---

# Manage enrichments in Console
> Enable, configure, and deploy enrichments for your pipelines using Snowplow Console UI.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/managing-enrichments/

Snowplow [Console](https://console.snowplowanalytics.com) enables you to manage enrichments that run on each of your pipelines and development environments.

## Recommended workflow

You can enable and edit enrichments on any pipeline, but we recommend the following workflow to test the configuration before deploying it:

- Navigate to a [development environment](/docs/testing/snowplow-micro/console/) (based on Snowplow Micro)
- Enable and configure the enrichment as required
- Send some events to the environment's endpoint and validate the fields appear as expected in the event payload
- Deploy the configuration to your pipeline(s)

Alternatively, you can use [Snowplow Micro](/docs/testing/snowplow-micro/local/) to test your changes locally.

> **Warning:** Invalid enrichment changes can halt data processing in the pipeline. For example, an enrichment might be pointing to a non-existent database file, or the JavaScript code in the JavaScript enrichment might have typos. This is why we recommend testing in a low-risk environment first.

## Configure an enrichment

Within [Console](https://console.snowplowanalytics.com), use the **Pipelines** section to navigate to the pipeline or development environment where you would like to change the enrichment. Open the **Enrichments** tab. You will see a listing of all enrichments and their current status (enabled or disabled).

Click on the enrichment you want to change. You will see the current configuration if the enrichment is enabled, and the default one if it's not. For each enrichment, the reference on the configuration format is linked in the top navigation bar.

- To enable or edit the enrichment, click on **Edit configuration**. Adjust the configuration and click **Publish** to deploy the changes. This will enable the enrichment if it was disabled.
- To disable the enrichment, click on the **⋮** menu and select **Disable**.

> **Warning:** Never put credentials, API keys, or salts directly in your enrichment configuration. For enrichments that require secrets (e.g., API Request enrichment, SQL Query, PII Pseudonymization), use the [Credentials Sharing Form](https://console.snowplowanalytics.com/contact-us) to share sensitive information with Snowplow. The Support team will place the secrets in secured environment variables on your behalf.

## Deploy an enrichment from a development environment to a pipeline

If you followed the recommended workflow to test an enrichment in a development environment, you will need to copy the same configuration into a pipeline.

To do that, navigate to the development environment under **Pipelines**, select the **Enrichments** tab and click on the enrichment in question. Then click **Deploy**, select the desired pipeline and click **Deploy configuration**.

---

# Manage enrichments in Snowplow Self-Hosted
> Enable and configure enrichments in Snowplow Self-Hosted deployments using Terraform on AWS, GCP, or Azure.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/managing-enrichments/terraform/

If you have installed Snowplow via [Quick Start](/docs/get-started/self-hosted/), you will have the following enrichments enabled by default:

- [UA parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)
- [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)
- [Campaign attribution](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)
- [Event fingerprint](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)
- [Referer parser](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)

To enable another enrichment, you'll need to provide additional configuration. Every enrichment has its own configuration schema, with required `name`, `vendor`, and `enabled` fields. Check the documentation for the enrichment you want to enable for details.

For example, to enable the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/), you'll need a JSON schema with this structure:

```json
schema: "iglu:com.snowplowanalytics.snowplow/anon_ip/jsonschema/1-0-1",

data: {

    name: "anon_ip",

    vendor: "com.snowplowanalytics.snowplow",

    enabled: true,

    parameters: {

        anonOctets: 1,

        anonSegments: 1

    }

}
```

The exact mechanism for providing this configuration depends on your cloud provider:

- Follow the instructions for [AWS](https://github.com/snowplow-devops/terraform-aws-enrich-kinesis-ec2#inserting-custom-enrichments), [GCP](https://github.com/snowplow-devops/terraform-google-enrich-pubsub-ce#inserting-custom-enrichments) or [Azure](https://github.com/snowplow-devops/terraform-azurerm-enrich-event-hub-vmss#inserting-custom-enrichments)
- Run `terraform apply`

> **Note:** The IAB and IP Lookups enrichments require a third-party database to function.

To disable any enrichment, follow the instructions for [AWS](https://github.com/snowplow-devops/terraform-aws-enrich-kinesis-ec2#disabling-default-enrichments), [GCP](https://github.com/snowplow-devops/terraform-google-enrich-pubsub-ce#disabling-default-enrichments) or [Azure](https://github.com/snowplow-devops/terraform-azurerm-enrich-event-hub-vmss#disabling-default-enrichments).

If you are using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing, you can configure enrichments following the [usage guide](/docs/testing/snowplow-micro/local/enrichments/).

---

# Configure your Snowplow pipeline
> Configure collector settings, manage enrichments, and secure your Snowplow data pipeline infrastructure.
> Source: https://docs.snowplow.io/docs/pipeline/

![Snowplow pipeline architecture diagram showing data flowing from web, mobile, server-side, pixel, and webhook sources through Collect, Validate, and Enrich stages. Validated events route to a Real-Time Good Stream and on to a data warehouse and BI tools; invalid events route to a Real-Time Bad Stream and Bad Logs.](/assets/images/architecture-16e4cd694721410bb8a1ff943a4c5255.png)

This section explains how to validate and update your [collector configuration](/docs/pipeline/collector/), manage [enrichments](/docs/pipeline/enrichments/), and has information on [security](/docs/pipeline/security/).

---

# Configure AWS KMS customer managed keys
> Set up AWS KMS custom encryption keys for enhanced pipeline security with full control over data encryption.
> Source: https://docs.snowplow.io/docs/pipeline/security/customer-managed-keys/

This guide provides step-by-step instructions for setting up AWS KMS (Key Management Service) custom keys for your Snowplow pipeline. Custom KMS keys provide enhanced security by allowing you to maintain full control over encryption keys used by your data infrastructure.

> **Note:** This is a bolt-on security feature available for enterprise customers using CDI Private Managed Cloud or CDI Cloud deployments.

AWS Custom KMS Keys ensure that all data stored at-rest in AWS services (Kinesis, SQS, and S3) are encrypted with a key that you own and control. This provides:

- Enhanced security: full control over encryption keys
- Compliance: meet regulatory requirements for data encryption
- Audit control: ability to revoke access to encrypted data
- Key management: centralized control over data encryption

## Prerequisites

To use custom KMS keys, you will need:

- AWS account with appropriate permissions to create and manage KMS keys
- Access to your AWS Console in the same region as your Snowplow pipeline
- Snowplow pipeline account ID (provided by your Customer Success Manager)
- Your security team approval for cross-account key sharing

This feature has some limitations:

- EMR cluster: if you're using the RDB Transformer component (only required for batch loading to Redshift and Databricks), it will need additional permissions adding (`kms:Decrypt`, `kms:GenerateDataKey*`) to enable access to customer S3 buckets with customer-managed KMS keys.
- Single key recommendation: while it's technically possible to use different keys for different services, we recommend using a single key for all pipeline resources for simplicity.
- Regional restrictions: keys must be created in the same region as your pipeline infrastructure.
- KMS API quota: BYOK operations consume additional KMS API calls beyond your baseline usage. Verify you have sufficient quota before enabling this feature.

## Step 1: Create your custom KMS key

Go to the AWS Console in the same region as your Snowplow pipeline and navigate to **KMS** → **Customer managed keys**. The URL will look similar to `https://us-east-1.console.aws.amazon.com/kms/home?region=us-east-1#/kms/keys`.

Click **Create key** and configure with the following settings:

| Setting             | Value                                            |
| ------------------- | ------------------------------------------------ |
| Key type            | Symmetric                                        |
| Key usage           | Encrypt and decrypt                              |
| Key material origin | KMS                                              |
| Regionality         | Single region                                    |
| Alias               | `snowplow-pipeline-key` (or your preferred name) |

When setting key administrators:

- Key administrators: select the roles/users from your security team that need administrative access to the key
- Other AWS Accounts: add your Snowplow pipeline account ID here

> **Note:** The "Other AWS Accounts" step is essential. Without adding the Snowplow pipeline account ID, the encryption will not work.

## Step 2: Key policy configuration

The key creation process will generate a policy similar to the one below. We strongly recommend allowing root access for the Snowplow pipeline account:

**Details**

```json
{

  "Id": "key-consolepolicy-3",

  "Version": "2012-10-17",

  "Statement": [

    {

      "Sid": "Enable IAM User Permissions",

      "Effect": "Allow",

      "Principal": {

        "AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:root"

      },

      "Action": "kms:*",

      "Resource": "*"

    },

    {

      "Sid": "Allow access for Key Administrators",

      "Effect": "Allow",

      "Principal": {

        "AWS": [

          "arn:aws:iam::YOUR_ACCOUNT_ID:role/YourSecurityRole",

          "arn:aws:iam::YOUR_ACCOUNT_ID:role/YourAdminRole"

        ]

      },

      "Action": [

        "kms:Create*",

        "kms:Describe*",

        "kms:Enable*",

        "kms:List*",

        "kms:Put*",

        "kms:Update*",

        "kms:Revoke*",

        "kms:Disable*",

        "kms:Get*",

        "kms:Delete*",

        "kms:TagResource",

        "kms:UntagResource",

        "kms:ScheduleKeyDeletion",

        "kms:CancelKeyDeletion",

        "kms:RotateKeyOnDemand"

      ],

      "Resource": "*"

    },

    {

      "Sid": "Allow use of the key",

      "Effect": "Allow",

      "Principal": {

        "AWS": "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:root"

      },

      "Action": [

        "kms:Encrypt",

        "kms:Decrypt",

        "kms:ReEncrypt*",

        "kms:GenerateDataKey*",

        "kms:DescribeKey"

      ],

      "Resource": "*"

    },

    {

      "Sid": "Allow attachment of persistent resources",

      "Effect": "Allow",

      "Principal": {

        "AWS": "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:root"

      },

      "Action": [

        "kms:CreateGrant",

        "kms:ListGrants",

        "kms:RevokeGrant"

      ],

      "Resource": "*",

      "Condition": {

        "Bool": {

          "kms:GrantIsForAWSResource": "true"

        }

      }

    }

  ]

}
```

Replace the placeholder values in this policy:

| Placeholder                    | Replace With                   | Example              |
| ------------------------------ | ------------------------------ | -------------------- |
| `YOUR_ACCOUNT_ID`              | Your AWS account ID            | `123456789012`       |
| `YourSecurityRole`             | Your security team's role name | `SecurityTeamRole`   |
| `YourAdminRole`                | Your administrative role name  | `AdminRole`          |
| `SNOWPLOW_PIPELINE_ACCOUNT_ID` | Snowplow pipeline account ID   | Provided by Snowplow |

If your security policies don't allow root access (`arn:aws:iam::<account_id>:root`), you can use these specific role patterns instead:

```json
{

  "Sid": "Allow use of the key - Specific Roles",

  "Effect": "Allow",

  "Principal": {

    "AWS": [

      "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:role/SnowplowOperator",

      "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:role/sp-*",

      "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:role/snowplow-*"

    ]

  },

  "Action": [

    "kms:Encrypt",

    "kms:Decrypt",

    "kms:ReEncrypt*",

    "kms:GenerateDataKey*",

    "kms:DescribeKey",

    "kms:CreateGrant",

    "kms:ListGrants",

    "kms:RevokeGrant"

  ],

  "Resource": "*"

}
```

Using non-root policies requires coordination with Snowplow Support to ensure all necessary roles are included. We strongly recommend the root access approach for simplicity.

## Step 3: Share key information with Snowplow

After creating your key, collect the following information:

- Key ARN: found in the key details page
  - Format: `arn:aws:kms:REGION:YOUR_ACCOUNT_ID:key/KEY_ID`
- Key ID: the unique identifier for your key
- Region: the AWS region where the key was created
- Environment name: e.g. `prod`, `qa1`

Share this information with your Snowplow Customer Success Manager or through a support ticket:

- Key ARN
- Environment name
- Confirmation that cross-account access is configured
- Which pipeline components you want encrypted (or confirm you want full pipeline encryption)
- Any specific requirements or restrictions

## Step 4: Snowplow configuration

Once you've shared the key information, Snowplow will configure the following pipeline components to use your custom KMS key.

All **pipeline S3 buckets** will be configured with your custom KMS key:

- Batch archive bucket
- Batch output bucket (used in RDB Transformer/Loader process for Redshift and Databricks)
- Batch processing bucket (used in RDB Transformer/Loader process for Redshift and Databricks)
- S3 bad events bucket
- S3 enriched events bucket

Pipeline **Kinesis streams** will be encrypted with your key:

- Good events stream
- Bad events stream
- Enriched events stream
- Incomplete events stream

If surge protection is enabled, **SQS queues** will use your custom key:

- Good events queue
- Bad events queue

**Other pipeline components** like Snowbridge will be granted access to decrypt/encrypt data using your key.

## Step 5: Post-configuration

After Snowplow applies your custom KMS key:

- Immediate effect: encryption settings change immediately on resources
- Temporary disruption: services may briefly fail until proper access is configured
- Drift application: Snowplow will apply necessary IAM policy updates to resolve access issues

You can verify the configuration by:

- Sending test events to your collector endpoint and confirming events appear in the enriched stream without errors
- Checking S3 bucket encryption settings in your AWS Console — objects should show your KMS key ARN under encryption properties
- Verifying Kinesis stream encryption status
- Confirming SQS queue encryption (if applicable)
- Reviewing **AWS CloudTrail logs** for `Decrypt` and `GenerateDataKey` operations against your key ID — the presence of these calls confirms your key is actively in use by the pipeline
- Monitoring pipeline health through Snowplow dashboards

## Rotating or changing your KMS key

If you need to switch to a different KMS key — for example, as part of a scheduled key rotation policy — follow these steps:

1. Create the new KMS key and configure its key policy as described in Steps 1 and 2 above.
2. Contact Snowplow Support with your environment name, the new Key ARN, and the current Key ARN being replaced.
3. Keep the old key active and accessible until Snowplow Support confirms the rotation is complete and all data encrypted with the old key has been processed and archived.

> **Danger:** Do not disable or delete the old KMS key until you receive explicit confirmation from Snowplow Support. Disabling the key prematurely while it is still referenced by your pipeline will cause data processing to stop immediately.

## Disabling customer-managed key encryption

> **Danger:** Disabling your KMS key while it is still referenced in the pipeline configuration will cause an immediate pipeline outage. Data processing will stop and stream latency will increase as events back up in queues.

If you need to disable customer-managed key encryption, you must follow this sequence — do not disable the key in AWS first:

1. Contact Snowplow Support and request that the KMS key references be removed from your pipeline configuration.
2. Wait for Snowplow to redeploy the updated configuration and confirm the key is no longer referenced.
3. Only after receiving confirmation from Snowplow Support, disable or delete the key in AWS.

## Security best practices

For recommended settings:

- Enable key rotation: turn on automatic key rotation for enhanced security
- Monitor usage: use AWS CloudTrail to track key usage
- Regular audits: periodically review key policies and access patterns
- Documentation: maintain internal documentation of key management procedures

For key management:

- Backup procedures: ensure key policies are backed up and version controlled
- Access reviews: regular review of who has administrative access to keys
- Incident response: plan for key compromise or rotation scenarios

## Troubleshooting

The following table covers the most common issues after enabling customer-managed key encryption.

| Issue                                        | Likely Cause                                                                 | Resolution                                                                                                            |
| -------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Data stops flowing after BYOK setup          | KMS key is unreachable or disabled                                           | Verify the key is enabled in the AWS KMS console and check CloudTrail for access denied errors                        |
| Some streams appear encrypted but not others | Snowplow did not apply the configuration to all pipeline stacks              | Contact Snowplow Support to verify all stacks have been updated                                                       |
| Access denied errors                         | Incorrect pipeline account ID in key policy, or key is in a different region | Verify the key policy includes the correct Snowplow pipeline account ID and that the key region matches your pipeline |

Pipeline failures after KMS implementation are expected temporarily while Snowplow applies the necessary IAM policy updates. Contact Support if issues persist beyond the expected configuration window.

To validate your key policy:

```bash
# Check current key policy

aws kms get-key-policy \

  --key-id YOUR_KEY_ID \

  --policy-name default
```

---

# Pipeline security
> Security features and best practices for protecting your Snowplow pipeline infrastructure and data.
> Source: https://docs.snowplow.io/docs/pipeline/security/

This section contains information relating to pipeline security.

## [📄️Customer managed keys](/docs/pipeline/security/customer-managed-keys/)

[Set up AWS KMS custom encryption keys for enhanced pipeline security with full control over data encryption.](/docs/pipeline/security/customer-managed-keys/)

---

# Use Signals in your application
> Consume Signals in your applications by retrieving calculated attributes on demand, or by subscribing to interventions that push actions in real time.
> Source: https://docs.snowplow.io/docs/signals/applications/

Once you've defined and published your [attributes](/docs/signals/attributes/) and [interventions](/docs/signals/interventions/), your applications can consume them in two ways:

- **Pull**: retrieve calculated attribute values from the Profiles Store on demand, for example, to personalize a page or provide context to a chatbot
- **Push**: subscribe to interventions, so that Signals notifies your application as soon as a user meets the trigger criteria

Both patterns require a [connection to Signals](/docs/signals/connection/) using the Python SDK, Node.js SDK, browser plugin, or API.

This section covers:

- [Services](/docs/signals/applications/services/): stable interfaces that group attribute group versions together for your applications, recommended for production use
- [Retrieve attributes](/docs/signals/applications/retrieve-attributes/): fetch calculated attribute values using a service or directly from an attribute group
- [Subscribe to interventions](/docs/signals/applications/subscribe/): receive intervention payloads and react to them in your application

---

# Retrieve attributes in your application
> Retrieve calculated attribute values from the Profiles Store using the Signals Python SDK, Node.js SDK, or API, either via a service or directly from an attribute group.
> Source: https://docs.snowplow.io/docs/signals/applications/retrieve-attributes/

Calculated attribute values are stored in the Profiles Store and consumed by your applications via the Python SDK, Node.js SDK, or API.

Start by [connecting to Signals](/docs/signals/connection/) to create a `Signals` client object.

## Use a service

The preferred way to retrieve attributes is via a [service](/docs/signals/concepts/#services), which lets you fetch attributes from multiple groups in one call. See [Services](/docs/signals/applications/services/) to define one using the Console or Python SDK.

**Python:**

Use `get_service_attributes()` to retrieve attributes from a service.

```python
calculated_values = sp_signals.get_service_attributes(

    name="my_service",

    attribute_key="domain_userid",

    identifier="218e8926-3858-431d-b2ed-66da03a1cbe5",

)
```

| Argument        | Description                                  | Type     | Required? |
| --------------- | -------------------------------------------- | -------- | --------- |
| `name`          | The name of the service                      | `string` | ✅         |
| `attribute_key` | The attribute key to retrieve attributes for | `string` | ✅         |
| `identifier`    | The specific attribute key value             | `string` | ✅         |

**Node.js:**

Use `getServiceAttributes()` to retrieve attributes for a single identifier, or `getBatchServiceAttributes()` for multiple identifiers in one call.

```typescript
const calculatedValues = await signals.getServiceAttributes({

  name: "my_service",

  attribute_key: "domain_userid",

  identifier: "218e8926-3858-431d-b2ed-66da03a1cbe5",

});
```

| Argument        | Description                                  | Type     | Required? |
| --------------- | -------------------------------------------- | -------- | --------- |
| `name`          | The name of the service                      | `string` | ✅         |
| `attribute_key` | The attribute key to retrieve attributes for | `string` | ✅         |
| `identifier`    | The specific attribute key value             | `string` | ✅         |

```typescript
const batchResults = await signals.getBatchServiceAttributes({

  name: "shopping_cart_service",

  attribute_key: "domain_userid",

  identifiers: [

    "218e8926-3858-431d-b2ed-66da03a1cbe5",

    "f47ac10b-58cc-4372-a567-0e02b2c3d479",

    "6ba7b810-9dad-11d1-80b4-00c04fd430c8"

  ]

});
```

| Argument        | Description                                  | Type       | Required? |
| --------------- | -------------------------------------------- | ---------- | --------- |
| `name`          | The name of the service                      | `string`   | ✅         |
| `attribute_key` | The attribute key to retrieve attributes for | `string`   | ✅         |
| `identifiers`   | Array of attribute key values to look up     | `string[]` | ✅         |

***

## Retrieve individual attributes

You can also retrieve attributes directly from a specific [attribute group](/docs/signals/concepts/#attribute-groups), without a service. This is useful when you want a small subset of attributes or haven't defined a service yet.

**Python:**

Use `get_group_attributes()` to retrieve specific attributes.

```python
calculated_values = sp_signals.get_group_attributes(

    name="my_attribute_group",

    version=1,

    attributes=["page_view_count"],

    attribute_key="domain_userid",

    identifier="218e8926-3858-431d-b2ed-66da03a1cbe5",

)
```

| Argument        | Description                             | Type                         | Required? |
| --------------- | --------------------------------------- | ---------------------------- | --------- |
| `name`          | The name of the attribute group         | `string`                     | ✅         |
| `version`       | The attribute group version             | `int`                        | ✅         |
| `attributes`    | The names of the attributes to retrieve | `string` or list of `string` | ✅         |
| `attribute_key` | The attribute key name                  | `string`                     | ✅         |
| `identifier`    | The specific attribute key value        | `string`                     | ✅         |

**Node.js:**

Use `getGroupAttributes()` to retrieve specific attributes from an attribute group.

```typescript
const calculatedValues = await signals.getGroupAttributes({

  name: "my_attribute_group",

  version: 1,

  attributes: ["page_view_count"],

  attribute_key: "domain_userid",

  identifier: "218e8926-3858-431d-b2ed-66da03a1cbe5",

});
```

| Argument        | Description                             | Type       | Required? |
| --------------- | --------------------------------------- | ---------- | --------- |
| `name`          | The name of the attribute group         | `string`   | ✅         |
| `version`       | The attribute group version             | `number`   | ✅         |
| `attributes`    | The names of the attributes to retrieve | `string[]` | ✅         |
| `attribute_key` | The attribute key name                  | `string`   | ✅         |
| `identifier`    | The specific attribute key value        | `string`   | ✅         |

***

---

# Define services
> Services group multiple versioned attribute groups into stable consumption interfaces for applications. Create and manage services to control which attribute versions your applications use.
> Source: https://docs.snowplow.io/docs/signals/applications/services/

[Services](/docs/signals/concepts/#services) group attribute groups together for serving to your applications. They provide a stable interface for [retrieving attributes](/docs/signals/applications/retrieve-attributes/): we strongly recommend using services in production applications.

**Console:**

To create a service, go to **Signals** > **Services** in Snowplow Console and follow the instructions.

![Create service form with name, description, owner, and attribute group selection](/assets/images/service-create-85585d669c129ebb7cb30d74209dfc12.png)

To configure a service, you'll need to specify:

- A unique name
- An optional description
- The email address of the primary owner or maintainer
- Which attribute groups to include

**Python SDK:**

Use the `Service` class to define a service. You can refer to attribute groups either by passing the objects directly, or by name and version using a dictionary:

```python
from snowplow_signals import Service



# Refer to attribute groups by name and version

my_service = Service(

    name='my_service',

    description='A collection of attribute groups',

    owner="user@company.com",

    attribute_groups=[

        {"name": "user_attributes", "version": 2},

        {"name": "session_attributes", "version": 1},

    ],

)



# Or pass the objects directly

my_service = Service(

    name='my_service',

    description='A collection of attribute groups',

    owner="user@company.com",

    attribute_groups=[

        my_attribute_group,

        another_attribute_group,

    ],

)
```

The table below lists all available arguments for a `Service`:

| Argument           | Description                                                  | Type                             | Required? |
| ------------------ | ------------------------------------------------------------ | -------------------------------- | --------- |
| `name`             | The name of the service                                      | `string`                         | ✅         |
| `description`      | A description of the service                                 | `string`                         | ❌         |
| `owner`            | The owner of the service                                     | `string`                         | ✅         |
| `attribute_groups` | List of attribute groups with optional version specification | list of `AttributeGroup` or dict | ❌         |

***

## Versioning

When choosing which attribute groups to include, you'll select a specific version of each attribute group.

Services themselves are not versioned. You can update them to use different attribute groups, or different attribute group versions, at any time.

**Console:**

![Service configuration showing selected attribute groups with version numbers](/assets/images/service-groups-adfabeee05def063578d69fda35203b2.png)

**Python SDK:**

Specify the version of each attribute group in the `attribute_groups` list using a dictionary with `name` and `version` keys, or by using a versioned `AttributeGroup` object directly.

***

## Manage services

**Console:**

Services are automatically published as soon as they're created. They're a wrapper for consuming calculated attributes, and don't cause any additional processing themselves.

To edit a service definition, go to the service details page and click the **Edit** button.

To delete a service, go to the service details page and click the `⋮` button, then choose **Delete**.

![Signals service details page in Snowplow Console showing the Edit button and the three-dot overflow menu expanded with the Delete option](/assets/images/service-edit-delete-7e8bf6d7c7571a961344c7d77b47263b.png)

**Python SDK:**

Use `publish()` to register services with Signals.

```python
from snowplow_signals import Signals



sp_signals = Signals({{ config }})



sp_signals.publish([

    my_service,

    my_other_service,

])
```

***

---

# Subscribe to and receive interventions in your applications
> Subscribe to interventions by attribute key ID to automatically respond to user behavior changes. Use the Python SDK, browser tracker plugin, or Signals API to receive intervention payloads.
> Source: https://docs.snowplow.io/docs/signals/applications/subscribe/

Subscribe to interventions to automatically respond within your application. You have three options for receiving interventions, depending on your use case or application:

- [Signals Python SDK](https://pypi.org/project/snowplow-signals/)
- Plugin for the [browser tracker](/docs/sources/web-trackers/)
- [Signals API](/docs/signals/connection/#signals-api)

Subscription is by attribute key ID, not by individual intervention. Start by [connecting to Signals](/docs/signals/connection/).

> **Note:** An intervention is sent only the first time the criteria are met. The [delivery example](#delivery-example) below shows how this works.

The subscription endpoint (`api/v1/interventions`) is public: it doesn't perform authentication or authorization, just checks the requested attribute keys. Knowing an attribute key ID grants access to its interventions. This is why ID values must be non-enumerable, such as canonically formatted UUIDs, so that they can't be guessed.

## Subscribe with the Python SDK

Subscribe by providing IDs for the attribute keys you're interested in receiving interventions for. The IDs must be in a non-enumerable format, such as UUIDs.

```python
from snowplow_signals import AttributeKeyIdentifiers, InterventionInstance



# Attribute keys to subscribe to

targets = AttributeKeyIdentifiers({

  "domain_sessionid": ["8c9104e3-c300-4b20-82f2-93b7fa0b8feb"],

  "domain_userid": ["218e8926-3858-431d-b2ed-66da03a1cbe5"],

})



# Define the subscription

subscription = sp_signals.pull_interventions(targets)



# Add a custom handler to be called with each intervention received

subscription.add_handler(print)



# Open the subscription request and begin fetching interventions as they are published

subscription.start()



# Block until an intervention is received

# Because of the `print` handler above, the intervention will be printed out

intervention_instance = subscription.get()



# Cancel the subscription and abort the connection/background thread

subscription.stop()
```

## Subscribe with the browser tracker plugin

For web applications using the Snowplow [browser tracker](/docs/sources/web-trackers/), you can subscribe to interventions using the [Signals browser plugin](https://www.npmjs.com/package/@snowplow/signals-browser-plugin).

The workflow is:

1. Create a Snowplow tracker with the plugin configured
2. Add custom handlers to react to the interventions
3. Subscribe to interventions

```typescript
import { newTracker } from '@snowplow/browser-tracker';

import {

  SignalsInterventionsPlugin,

  addInterventionHandlers,

  subscribeToInterventions,

} from '@snowplow/signals-browser-plugin';



// Install the Signals Intervention plugin

newTracker('sp1', '{{collector_url}}', {

  appId: 'my-app-id',

  plugins: [ SignalsInterventionsPlugin() ],

});



// Add custom handlers

addInterventionHandlers({

  myHandler(intervention) {

    console.log("intervention received!", intervention);

  },

});



// Subscribe to interventions using your Signals API endpoint

subscribeToInterventions({

  endpoint: "000000000000.signals.snowplowanalytics.com",

});
```

By default, the plugin will automatically subscribe to interventions for the `domain_userid` and `domain_sessionid` attribute keys.

You can optionally configure the plugin for additional attribute keys. There are two options:

- `attributeKeyTargets`: dynamically subscribe to attribute key IDs extracted from the tracker's generated events
- `attributeKeyIds`: subscribe to a specific attribute key ID

This example assumes you've defined a `pageview_id` attribute key, based on the `web_page` entity added by default to all web events.

```typescript
subscribeToInterventions({

  endpoint: "000000000000.signals.snowplowanalytics.com", // Signals API endpoint

  attributeKeyTargets: {

    pageview_id: "/co/com.snowplowanalytics.snowplow/web_page/id",

  },

  attributeKeyIds: {

    network_userid: "177234df-d421-412e-ad8d-8bf97515b280",

  },

});
```

The plugin will automatically disconnect/reconnect whenever the attribute key IDs change. The browser will close the connection on navigation away from the page.

### Intervention events

The plugin generates Snowplow events to track interventions. The events include the intervention payload as a custom entity.

- [`iglu:com.snowplowanalytics.signals/intervention_receive/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.signals/intervention_receive/jsonschema/1-0-0): tracked when the plugin receives an intervention
- [`iglu:com.snowplowanalytics.signals/intervention_handle/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.signals/intervention_handle/jsonschema/1-0-0): tracked when a custom handler is passed the intervention payload and reports successful handling (does not return an error)
- [`iglu:com.snowplowanalytics.signals/intervention_handle_error/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.signals/intervention_handle_error/jsonschema/1-0-0): tracked when a custom handler is passed the intervention payload and reports failure (throws an error)

## Payload

When delivered, interventions contain the following information:

| Field                       | Description                                                             | Type      |
| --------------------------- | ----------------------------------------------------------------------- | --------- |
| `intervention_id`           | A unique identifier for this triggered intervention                     | `string`  |
| `name`                      | The unique name/identifier of the intervention                          | `string`  |
| `version`                   | The intervention version                                                | `integer` |
| `attributes`                | An object containing the target's attributes on intervention triggering | `object`  |
| `target_attribute_key`      | An object containing the target attribute key information               | `object`  |
| `target_attribute_key.name` | The name of the target attribute key                                    | `string`  |
| `target_attribute_key.id`   | The value of the target attribute key                                   | `string`  |

Here's an example intervention payload:

```json
{

  "intervention_id": "550e8400-e29b-41d4-a716-446655440000",

  "name": "high_value_cart_abandonment",

  "version": 2,

  "attributes": {

    "cart_value": 250.00,

    "items_in_cart": 3,

    "time_on_checkout_page": 120,

    "previous_purchases": 8,

    "last_activity_timestamp": "2025-09-23T14:30:15Z"

  },

  "target_attribute_key": {

    "name": "domain_userid",

    "id": "218e8926-3858-431d-b2ed-66da03a1cbe5"

  }

}
```

## Delivery example

This example shows how targeting, subscription, and the send-once rule work together. Consider these two interventions:

- `intervention1` targets the `domain_userid` and `domain_sessionid` attribute keys

  - It has two criteria rules, where `any` must be met
  - The first rule is: `attribute_group1:page_view_count == 10`
  - The second rule is: `attribute_group2:ad_count is not null`

- `intervention2` targets the `domain_userid` attribute key
  - It has one criteria rule, `attribute_group3:button_click_count > 20`

To receive interventions for the current user and session, subscribe to their interventions by providing the `domain_userid` and `domain_sessionid` ID values.

Interventions will be triggered when:

- The user views 10 pages, **or** the session has an ad event

  - `intervention1` will be delivered to one of the subscribed targets.
  - Signals will initially create a payload for each target. However, they'll have the same ID as they're triggered by a single event, and will be deduped.

- The user exceeds 20 button clicks for the first time
  - `intervention2` will be delivered to the subscribed user.

Assuming it's the user's first session, the flow looks like this:

- First page view

- First ad event
  - `intervention1` is triggered and delivered to one of the targets

- More page views, button clicks, and ad events

- Tenth page view
  - Nothing happens: because `intervention1` sent already on seeing the ad event in this session, it's not triggered again

- Twenty-first button click

  - The attribute value changes from 20 to 21, crossing the required threshold
  - `intervention2` is triggered and delivered to the user target

- Twenty-second button click
  - Nothing happens: the threshold has already passed

This user has already seen more than 10 pages, so `intervention1` can never be triggered by that rule for them. However, if their current session expires, and the application subscribes to their new `domain_sessionid` ID, `intervention1` can be triggered again by the first ad event of the session.

Likewise, the user has already clicked a button more than 20 times, so `intervention2` will never be sent to them again.

---

# Define attribute groups
> Define attribute groups to calculate behavioral data from real-time event streams or sync pre-calculated warehouse tables. Configure attributes, attribute keys, TTL lifetimes, and test definitions before publishing.
> Source: https://docs.snowplow.io/docs/signals/attributes/attribute-groups/

Define the behavior you want to capture in [attribute groups](/docs/signals/concepts/#attribute-groups). Choose whether to calculate attributes from your event stream or sync pre-calculated values from your warehouse.

**Console:**

To create an attribute group, go to **Signals** > **Attribute groups** in Snowplow Console and follow the instructions.

![Create attribute group form with name, description, data source, and owner fields](/assets/images/attribute-group-create-142d5efd566fe504028a4cca6d7c92e5.png)

The first step is to specify:

- A unique name
- An optional description
- The email address of the primary owner or maintainer
- Which data source you want to use

**Python SDK:**

Use a `StreamAttributeGroup` to calculate attributes from the real-time event stream.

```python
from snowplow_signals import StreamAttributeGroup, domain_sessionid



my_stream_attribute_group = StreamAttributeGroup(

    name="my_stream_attribute_group",

    version=1,

    attribute_key=domain_sessionid,

    owner="user@company.com",

    attributes=[

        # Previously defined attributes

        page_view_count,

        products_added_to_cart_feature,

    ],

)
```

***

## Define the attribute group

An attribute group definition has several parts, described in this section. The most important choice is the data source, which determines how attributes are calculated and which options are available.

### Data source

There are two [sources](/docs/signals/concepts/#data-sources) to choose from:

**Console:**

- **Stream**: real-time Snowplow event stream, with an optional warehouse backfill
- **Warehouse**: pre-calculated values in a warehouse table that you sync to the Profiles Store

**Python SDK:**

- **`StreamAttributeGroup`**: calculates attributes from the real-time event stream, with an optional backfill
- **`ExternalBatchAttributeGroup`**: syncs pre-calculated values from an existing warehouse table to the Profiles Store

***

Attribute groups are configured differently based on the data source.

### Stream

Signals calculates attributes from events in your real-time stream. Check out the [quick start tutorial](/tutorials/signals-quickstart/start/) for a step-by-step guide.

You'll need to define the [attributes](/docs/signals/attributes/attributes/) you want to calculate from your event stream.

#### Backfill attributes

Stream attribute groups only calculate attributes from the moment they are published. If you want to include historical data, enable backfill when creating the group.

**Console:**

> **Note:** A warehouse connection is required to use the backfill option. Only Snowflake and BigQuery are supported currently.

Enable **Backfill attributes** when creating the group. A date picker appears — select the date from which Signals should backfill attribute values from your Snowplow `atomic` events table. On publish, Signals backfills all events from that date up to the publish timestamp using your warehouse.

Backfill operates within a defined period: from your selected start date to the publish timestamp. This can take some time depending on data volume. The streaming engine starts immediately on publish and processes all new events in real time. Once backfill completes its defined period, only the streaming engine is active. You can monitor progress with the status bar in Snowplow Console.

![Stream attribute group with Backfill attributes toggle enabled and date picker shown](/assets/images/attribute-group-stream-backfill-9b8122806062098ae1f74ec27dea6856.png)

**Python SDK:**

Set `backfill_since_tstamp` on your `StreamAttributeGroup` to specify the earliest date to backfill from.

```python
from datetime import datetime, timezone

from snowplow_signals import StreamAttributeGroup, domain_userid



my_stream_attribute_group = StreamAttributeGroup(

    name="my_stream_attribute_group",

    version=1,

    attribute_key=domain_userid,

    owner="user@company.com",

    attributes=[page_view_count],

    backfill_since_tstamp=datetime(2025, 1, 1, tzinfo=timezone.utc),

)
```

***

### Warehouse

Attribute groups with a warehouse source don't require attribute definition, as no calculation is performed. This source type syncs existing, pre-calculated warehouse values to your Profiles Store using the batch engine.

See [Warehouse configuration](/docs/signals/attributes/warehouse-config/) for full configuration details.

**Console:**

Provide the warehouse and table details, and select which fields you want to send to Signals.

![Warehouse source configuration showing warehouse table and field mapping options](/assets/images/attribute-group-warehouse-fields-645921d8d99b1a8c4cd7aec3ac157b2a.png)

**Python SDK:**

Use `ExternalBatchAttributeGroup` instead of `StreamAttributeGroup`. See [Warehouse configuration](/docs/signals/attributes/warehouse-config/) for the full SDK reference.

***

### Attribute group options

**Console:**

Configure the attribute group fields described at the top of this page. Additional options such as backfill and TTL are covered in the sections below.

**Python SDK:**

The table below lists all available arguments for `StreamAttributeGroup`:

| Argument                | Description                                           | Type                | Default | Required? |
| ----------------------- | ----------------------------------------------------- | ------------------- | ------- | --------- |
| `name`                  | The name of the attribute group                       | `string`            |         | ✅         |
| `version`               | The version of the attribute group                    | `int`               | 1       | ❌         |
| `attribute_key`         | The attribute key associated with the attribute group | `AttributeKey`      |         | ✅         |
| `owner`                 | The owner of the attribute group                      | `Email`             |         | ✅         |
| `description`           | A description of the attribute group                  | `string`            |         | ❌         |
| `ttl`                   | Time-to-live for attributes in the Profiles Store     | `timedelta`         |         | ❌         |
| `attributes`            | List of attributes to calculate                       | list of `Attribute` |         | ✅         |
| `online`                | Calculate attributes (`True`) or not (`False`)        | `bool`              | `True`  | ❌         |
| `backfill_since_tstamp` | How far in time to backfill from                      | `datetime`          |         | ❌         |

Use the `online` property to control whether Signals should actively compute the attributes, or just register the configuration. If you only want to publish the attribute group definitions without calculating attribute values yet, set `online=False`.

***

### Attribute keys

All attribute groups need an [attribute key](/docs/signals/concepts/#attribute-keys). See [Attribute keys](/docs/signals/attributes/attribute-keys/) for details on using built-in keys and creating custom ones.

### Attribute lifetimes

TTL (time-to-live) configuration applies to attributes without a period. For attributes with a period, the TTL is set automatically to match the period and any configured TTL is ignored.

We recommend setting a TTL on your attribute group to avoid stale values persisting in your Profiles Store indefinitely. The defaults are 7 days for stream attribute groups and 365 days for warehouse attribute groups. If no TTL is set on the attribute group, the attribute key's TTL is used; if neither is set, the defaults apply.

When an attribute has not been updated for its defined TTL, its value is deleted: fetching it will return a `None` value. If Signals then processes a new event that updates the attribute, or syncs new data from the warehouse, the expiration timer is reset.

**Console:**

Configure a TTL when creating or updating the attribute group.

![Attribute group TTL configuration field](/assets/images/ttl-7c4ffac39d6a364e22006def22f006ef.png)

**Python SDK:**

Set `ttl` on your attribute group using a `timedelta`:

```python
from datetime import timedelta

from snowplow_signals import StreamAttributeGroup, user_id



stream_attribute_group = StreamAttributeGroup(

    name="comprehensive_stream_attribute_group",

    version=2,

    attribute_key=user_id,

    owner="data-team@company.com",

    attributes=[page_view_count, session_duration],

    ttl=timedelta(days=90),

)
```

***

## Test attribute definitions

After defining one or more [attributes](/docs/signals/attributes/attributes/) for stream attribute groups, you can test the configuration before publishing. This outputs a table of attributes calculated from your `atomic` events table, using a random subset of events from the last hour.

> **Note:** A warehouse connection is required to test attribute definitions.

**Console:**

Click the **Run preview** button in the bottom right corner.

![Run preview button in the bottom right corner of the attribute group editor](/assets/images/run-preview-661b2186039833a97536d50d7419d3d6.png)

**Python SDK:**

Use the `test()` method.

```python
from snowplow_signals import Signals



sp_signals = Signals({{ config }})



test_data = sp_signals.test(

    attribute_group=my_attribute_group,

    app_ids=["website"]

)
```

To inspect an existing attribute group's definition, use `get_attribute_group()`:

```python
attribute_definitions = sp_signals.get_attribute_group(

    name="my_attribute_group",

    version=1,

)

print(attribute_definitions)
```

| Argument  | Description                     | Type     | Required? |
| --------- | ------------------------------- | -------- | --------- |
| `name`    | The name of the attribute group | `string` | ✅         |
| `version` | The attribute group version     | `int`    | ❌         |

If you don't specify a version, Signals retrieves the latest version.

***

## Publish attribute groups

**Console:**

Click **Create attribute group** to save it as a draft.

![Draft attribute group page showing Edit and Publish buttons](/assets/images/attribute-group-draft-c1d87d4475afae4157d9f077bd4b0d77.png)

Click **Edit** to make changes. To send the configuration to your Signals infrastructure, click **Publish**. This allows Signals to start calculating attributes or syncing tables, and populating the Profiles Store.

![Published attribute group page showing active status and management options](/assets/images/attribute-group-published-2cf1af60f705a14cbe0ba0d61c5390f6.png)

**Python SDK:**

Use `publish()` to register attribute groups with Signals.

```python
from snowplow_signals import Signals



sp_signals = Signals({{ config }})



sp_signals.publish([

    my_attribute_group,

    my_other_attribute_group,

])
```

***

Once published, Signals starts calculating attribute values and storing them in the Profiles Store. To use them, [retrieve the attributes in your application](/docs/signals/applications/retrieve-attributes/), typically via a [service](/docs/signals/applications/services/).

## Versioning

Attribute groups are versioned, which allows you to iterate on definitions without breaking downstream processes. You'll select specific attribute group versions when you define [services](/docs/signals/applications/services/).

**Console:**

All attribute groups start as `v1`. If you make changes to the definition, the version is automatically incremented when you edit and republish.

**Python SDK:**

Use `version=1` for the first version. After publishing, if you want to change the definition, increment the version number manually.

***

## Delete an attribute group

**Console:**

To unpublish or delete an attribute group, click the `⋮` button on the group details page.

![Attribute group management menu showing Edit, Unpublish, and Delete options](/assets/images/attribute-group-edit-delete-cdff6c24c74fe31d67ad224c2c79f5cd.png)

Unpublishing is version specific. It stops Signals from calculating attributes for that version, but existing attribute values remain in your Profiles Store. You can republish it later if needed.

Choose **Delete** to permanently delete all versions of the attribute group, along with attribute values in your Profiles Store for this group.

If the attribute group version is used by a [service](/docs/signals/concepts/#services), you'll need to update the service definition before unpublishing or deleting.

If the attribute group version is used by a published [intervention](/docs/signals/concepts/#interventions), deleting or unpublishing it will unpublish the intervention.

**Python SDK:**

Use `unpublish()` to stop active calculation without losing the attribute group definition. You can republish it later if needed.

Use `delete()` to permanently remove the attribute group and its calculated attribute values from the Profiles Store. You must unpublish before deleting.

```python
# Unpublish

sp_signals.unpublish([my_attribute_group])



# Delete permanently (must unpublish first)

sp_signals.delete([my_attribute_group])
```

***

---

# Define attribute keys
> Create custom attribute keys based on any event property to specify the analytical context for attribute calculations. Use built-in keys like domain_userid or define your own.
> Source: https://docs.snowplow.io/docs/signals/attributes/attribute-keys/

An [attribute key](/docs/signals/concepts/#attribute-keys) is the identifier that attributes are calculated against. All attribute groups need an attribute key.

Signals includes four built-in attribute keys, based on commonly used identifiers from the atomic [user-related fields](/docs/fundamentals/canonical-event/#user-fields) in all Snowplow events.

**Console:**

The built-in attribute keys are available to select when creating an attribute group in Snowplow Console: `domain_userid`, `domain_sessionid`, `network_userid`, and `user_id`.

**Python SDK:**

Import the built-in attribute keys directly:

```python
from snowplow_signals import (

    domain_userid,

    domain_sessionid,

    user_id,

    network_userid,

)
```

***

## Custom attribute keys

You can define custom attribute keys to calculate attributes against identifiers other than the built-in ones. How you specify the key depends on the attribute group [data source](/docs/signals/concepts/#data-sources).

For stream attribute groups, the key can be any property in your events:

- [Atomic](/docs/fundamentals/canonical-event/#common-fields) properties: fields of the core Snowplow event, available for all events
- Event schema properties: properties within a [self-describing event](/docs/fundamentals/events/#self-describing-events)
- Entity properties: properties from schemas tracked as [entities](/docs/fundamentals/entities/)

For [warehouse attribute groups](/docs/signals/attributes/warehouse-config/), the attributes are pre-calculated in your warehouse table, so the key isn't based on event properties. Instead, define the attribute key using the name of the table column that contains the key values.

**Console:**

Navigate to **Signals** > **Attribute keys** in Console. Click the **Create attribute key** button.

![Create attribute key form with name, description, and property selection](/assets/images/attribute-key-create-7771bedf4e1b02a0915a80001741cef5.png)

You will need to provide:

- A unique name
- An optional description
- An optional email address for the primary owner or maintainer
- Which property you want to calculate attributes against: an atomic field, or a property from an event or entity schema
  - For keys used with warehouse attribute groups, provide the name of the table column that contains the key values instead

To edit or delete a custom attribute key, go to the key details page and click the **Edit** button, or the `⋮` button followed by **Delete**.

**Python SDK:**

Use the `AttributeKey` class to define a custom attribute key. For stream attribute groups, specify the property with the same [property classes](/docs/signals/attributes/attributes/#select-a-property) used for defining attributes: `AtomicProperty`, `EventProperty`, or `EntityProperty`.

```python
from snowplow_signals import AttributeKey, AtomicProperty



app_id_attribute_key = AttributeKey(

    name="app_id_attribute_key",

    description="The id for the app",

    property=AtomicProperty(name="app_id"),

)
```

The table below lists all available arguments:

| Argument          | Description                                                                                     | Type                                                   | Required?                              |
| ----------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------ | -------------------------------------- |
| `property`        | The property to calculate attributes against, for stream attribute groups                       | `AtomicProperty`, `EventProperty`, or `EntityProperty` | Either `property` or `external_column` |
| `external_column` | The column in your warehouse table that contains the key values, for warehouse attribute groups | `string`                                               | Either `property` or `external_column` |
| `name`            | The name of the attribute key (derived from `property` or `external_column` if not set)         | `string`                                               | ❌                                      |
| `description`     | A description of the attribute key                                                              | `string`                                               | ❌                                      |
| `owner`           | The owner of the attribute key                                                                  | `string`                                               | ❌                                      |
| `ttl`             | Time attributes for this key will live in the Profiles Store                                    | `timedelta`                                            | ❌                                      |

> **Note:** Earlier versions of the SDK used a `key` argument that accepted atomic properties only. It's deprecated: use `property` instead.

Here's a full example using a property from a custom `user` entity:

```python
from datetime import timedelta

from snowplow_signals import AttributeKey, EntityProperty



customer_tier_attribute_key = AttributeKey(

    name="customer_tier_attribute_key",

    description="Attribute key for analyzing behavior by customer tier",

    property=EntityProperty(

        vendor="com.example",

        name="user",

        major_version=1,

        path="tier",

    ),

    owner="analytics-team@company.com",

    ttl=timedelta(days=365),

)
```

For [warehouse attribute groups](/docs/signals/attributes/warehouse-config/), use `external_column` to define the key from the table column that contains the key values:

```python
from snowplow_signals import AttributeKey



customer_id_attribute_key = AttributeKey(

    external_column="CUSTOMER_ID",

    description="Attribute key for the customer ID column in the transactions table",

)
```

The key's `name` is set to the column name automatically. If you set both, they must match.

Custom attribute keys are used in attribute groups in the same way as built-in keys.

***

---

# Define attributes
> Define individual attributes within attribute groups by selecting event schemas or event specifications, properties, and aggregation types. Use time periods and criteria filters to control when attributes update.
> Source: https://docs.snowplow.io/docs/signals/attributes/attributes/

[Attributes](/docs/signals/concepts/#attribute-groups) are defined as part of attribute groups. To create an attribute, you'll need to set:

- A name that describes the attribute
- Which event schema or event specification to calculate it from
- What property in the schema to consider for the calculation
- What kind of aggregation you want to calculate over time, e.g. `mean` or `last`

Attribute calculation starts when the definitions are published, and values are not backdated.

## Configure the attribute

Every attribute needs [events](#select-events) to calculate from, and an [aggregation](#choose-an-aggregation) to calculate. Most aggregations also operate on a [property](#select-a-property) of those events. The [time period](#set-a-time-period) and [criteria](#filter-with-criteria) settings are optional refinements.

In Console, open your attribute group and use the attribute configuration interface to fill in the fields. The time period and criteria settings are within **More options**.

![Attribute configuration interface showing name, event filter, and aggregation options](/assets/images/attribute-group-attributes-112be3abdd020aa9bf10977a7b68fb1c.png)

With the Python SDK, these settings are arguments to the `Attribute` class, listed in [all attribute options](#all-attribute-options).

### Select events

Use the event filter to choose which event type to calculate the attribute from.

**Console:**

![Event filter dropdown showing available event specifications, Snowplow, and custom event schemas](/assets/images/attribute-event-specs-6a0d11cd44d222a129e9ff120c06bcab.png)

Click the dropdown to see the available events, listed by name and vendor:

- **Event Specifications**: select any [event specification](/docs/event-studio/tracking-plans/event-specifications/) from an existing [tracking plan](/docs/event-studio/tracking-plans/).
- **Snowplow events**: select any built-in Snowplow or [Iglu Central](https://iglucentral.com) schema. For legacy reasons, to calculate an attribute from [structured](/docs/events/custom-events/#structured-events) events find `event (com.google.analytics.measurement-protocol)`.
- **Custom events**: select any schema or data structure available within your pipeline.

> **Note:** The search finds direct matches only, so use the exact name of the event, schema, or vendor.

Once you've selected an event and version, click **Confirm** to add the attribute to your attribute group.

**Python SDK:**

The `events` list describes the types of events the attribute is calculated from, as references to Snowplow event schemas.

An `Event` accepts the following parameters:

| Argument  | Description                                     | Type     |
| --------- | ----------------------------------------------- | -------- |
| `name`    | `event_name` column in `atomic.events` table    | `string` |
| `vendor`  | `event_vendor` column in `atomic.events` table  | `string` |
| `version` | `event_version` column in `atomic.events` table | `string` |

All parameters are optional and work as wildcards. Some examples:

```python
# A specific event version

Event(

    name="destination",

    vendor="com.snowplowanalytics.snowplow.media",

    version="2-0-2"

)



# All versions of an event

Event(

    name="destination",

    vendor="com.snowplowanalytics.snowplow.media",

)



# All events for a vendor

Event(vendor="com.snowplowanalytics.snowplow.media")



# Built-in Snowplow events

sp_page_view = Event(name="page_view", vendor="com.snowplowanalytics.snowplow", version="1-0-0")

sp_page_ping = Event(name="page_ping", vendor="com.snowplowanalytics.snowplow", version="1-0-0")

# Structured events

sp_structured = Event(name="event", vendor="com.google.analytics", version="1-0-0")
```

***

### Choose an aggregation

Signals supports the following aggregation types:

| Aggregation           | Description                                                                                                             | Required property type in schema      |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| Counter               | Count events                                                                                                            | No property used for this aggregation |
| Sum                   | Sum of property values                                                                                                  | Numeric                               |
| Min                   | Minimum property value                                                                                                  | Numeric                               |
| Max                   | Maximum property value                                                                                                  | Numeric                               |
| Mean                  | Average of property values                                                                                              | Numeric                               |
| First                 | First property value seen                                                                                               | String, Numeric, Boolean              |
| Last                  | Last property value seen                                                                                                | String, Numeric, Boolean              |
| Most Frequent         | Most frequent property value seen                                                                                       | String, Numeric, Boolean              |
| Least Frequent        | Least frequent property value seen                                                                                      | String, Numeric, Boolean              |
| Approx Count Distinct | Approximate distinct count ([HyperLogLog](https://redis.io/docs/latest/develop/data-types/probabilistic/hyperloglogs/)) | String, Numeric, Boolean              |
| Category Count        | Dictionary of unique values and their counts                                                                            | String, Numeric, Boolean              |
| Unique List           | List of unique property values                                                                                          | String, Numeric, Boolean              |

A property isn't used for `counter` aggregation. To only count events with a specific property value, use a criteria filter.

**Console:**

Select the aggregation type from the dropdown in the attribute configuration interface.

**Python SDK:**

Set the `aggregation` argument using the lowercase snake\_case version of the aggregation name, e.g. `"counter"`, `"most_frequent"`, `"approx_count_distinct"`.

***

### Select a property

You can calculate attributes based on properties in any part of your events:

- [Atomic](/docs/fundamentals/canonical-event/) properties: available for all events
- Event schema properties: properties within your chosen event
- Entity properties: properties from schemas tracked as entities with your chosen event

**Console:**

![Property selection interface with atomic, event schema, and entity property tabs](/assets/images/attribute-property-selector-d2cb6ca2f9acc7207cf1a5eaa5c35f9c.png)

Click **Confirm** to specify the property for this attribute.

**Python SDK:**

Use the `property` argument on `Attribute` with one of these helper classes:

- `AtomicProperty` — targets atomic properties in the event payload
- `EventProperty` — targets properties in the event data structure
- `EntityProperty` — targets properties in entity data structures

```python
# Atomic property

AtomicProperty(name="app_id")



# Property in an event data structure

EventProperty(

    vendor="com.example",

    name="test_event",

    major_version=1,

    path="action"

)



# Property in an entity

EntityProperty(

    vendor="com.example",

    name="user_context",

    major_version=1,

    path="age"

)
```

***

### Set a time period

Add an optional time period to aggregate over a rolling window. Signals won't include events older than the specified time period in the calculation.

**Console:**

Find the time period option within **More options**. Click **Done** to save it.

![Time period configuration dialog for setting rolling window attributes](/assets/images/attribute-set-period-1245a531daa2acb8b7019decfda6c451.png)

**Python SDK:**

Set `period` on your `Attribute` using a Python `timedelta`:

```python
from datetime import timedelta



my_attribute = Attribute(

    ...,

    period=timedelta(minutes=10),

)
```

***

### Filter with criteria

Use criteria to filter the events used to calculate an attribute. They allow you to be specific about which subsets of events should trigger attribute updates. For example, instead of counting all page views in a user's session, you may wish to calculate only views for the homepage, or a login page.

**Console:**

Find the criteria option within **More options**.

Defining criteria has three steps:

1. Select which property to filter on, similarly to the property selection for the attribute
2. Choose which logical operator to use
3. Enter the value to filter on

If you enter multiple criteria, you will have the option to require `all` or `any` of them to be met for the attribute to update.

![Criteria filter configuration with property selection, operator, and value fields](/assets/images/attribute-criteria-8db2a31c1ee6c759463b7dc0176fb9fb.png)

Click **Done** to save the criteria when you're finished.

**Python SDK:**

The `criteria` argument takes a `Criteria` object, which contains `Criterion` conditions.

| Argument | Description                        | Type                |
| -------- | ---------------------------------- | ------------------- |
| `all`    | All conditions must be met         | list of `Criterion` |
| `any`    | At least one condition must be met | list of `Criterion` |

Use `Criterion` operator methods to define filtering rules:

- `.eq` — equality (`=`)
- `.neq` — non-equality (`!=`)
- `.gt` — greater than
- `.gte` — greater than or equal to
- `.lt` — less than
- `.lte` — less than or equal to
- `.like` — pattern match (`LIKE`)
- `.in_list` — value in list (`IN`)

For example, to calculate an attribute for page views of either the FAQs or contact page:

```python
from snowplow_signals import Criteria, Criterion, AtomicProperty



criteria = Criteria(

    any=[

        Criterion.like(AtomicProperty(name="page_url"), "%/faq%"),

        Criterion.like(AtomicProperty(name="page_url"), "%/contact-us%"),

    ]

)
```

***

## All attribute options

The table below lists all available arguments for a Python SDK `Attribute`. The same options are available in the Console attribute configuration interface.

| Argument        | Description                                                   | Type                                                                                                                                                                                                                       | Required? |
| --------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `name`          | The name of the attribute                                     | `string`                                                                                                                                                                                                                   | ✅         |
| `description`   | The description of the attribute                              | `string`                                                                                                                                                                                                                   | ❌         |
| `events`        | List of Snowplow `Event`s to calculate the attribute from     | list of `Event`                                                                                                                                                                                                            | ✅         |
| `aggregation`   | The calculation to perform                                    | one of: `counter`, `sum`, `min`, `max`, `mean`, `first`, `last`, `most_frequent`, `least_frequent`, `approx_count_distinct`, `category_count`, `unique_list`                                                               | ✅         |
| `type`          | The type of the aggregation result                            | one of: `bytes`, `string`, `int32`, `int64`, `double`, `float`, `bool`, `dict`, `unix_timestamp`, `bytes_list`, `string_list`, `int32_list`, `int64_list`, `double_list`, `float_list`, `bool_list`, `unix_timestamp_list` | ✅         |
| `criteria`      | Filters to apply to events                                    | `Criteria`                                                                                                                                                                                                                 | ❌         |
| `property`      | The property of the event or entity to use in the aggregation | `string`                                                                                                                                                                                                                   | ❌         |
| `period`        | The time window over which to calculate the aggregation       | `timedelta`                                                                                                                                                                                                                | ❌         |
| `default_value` | Default value if aggregation returns no results               |                                                                                                                                                                                                                            | ❌         |

## Examples

The following examples show complete attribute configurations, using the Python SDK.

### Minimal example

This is the minimum configuration needed to create an attribute:

```python
from snowplow_signals import Attribute, Event



my_attribute = Attribute(

    name="button_click_counter",

    type="int32",

    events=[

        Event(

            vendor="com.snowplowanalytics.snowplow",

            name="button_click",

            version="1-0-0",

        )

    ],

    aggregation="counter"

)
```

Once applied and active, this definition triggers every time Signals processes an event with the schema `iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0`. The stored attribute starts at 0 and increases by 1 with every `button_click` event.

### Filtered counter with a time window

Count `button_click` events only where the button `id` is `generate_emoji_btn`, over a rolling 10-minute window.

```python
from snowplow_signals import Attribute, Event, Criteria, Criterion, EventProperty

from datetime import timedelta



button_click_counter_attribute = Attribute(

    name="emoji_button_click_counter",

    description="The number of clicks for the 'generate emoji' button",

    type="int32",

    events=[

        Event(

            vendor="com.snowplowanalytics.snowplow",

            name="button_click",

            version="1-0-0",

        )

    ],

    aggregation="counter",

    criteria=Criteria(

        all=[

            Criterion.eq(

                EventProperty(

                    vendor="com.snowplowanalytics.snowplow",

                    name="button_click",

                    major_version=1,

                    path="id"

                ),

                "generate_emoji_btn"

            )

        ]

    ),

    period=timedelta(minutes=10),

    default_value=0

)
```

### Last atomic property value

Track the most recent referrer source using the `mkt_medium` atomic property, calculated from either a page view or a custom event.

```python
from snowplow_signals import Attribute, Event, AtomicProperty



referrer_source_attribute = Attribute(

    name="referrer_source",

    description="Referrer",

    type="string",

    events=[

        Event(name="page_view", vendor="com.snowplowanalytics.snowplow", version="1-0-0"),

        Event(name="login_landing", vendor="com.business.example", version="1-0-0"),

    ],

    aggregation="last",

    property=AtomicProperty(name="mkt_medium"),

)
```

### Sum of an entity property

Sum the `price` of product entities in ecommerce events, counting only `transaction` events.

```python
from snowplow_signals import Attribute, Event, Criteria, Criterion, EntityProperty, EventProperty



my_new_attribute = Attribute(

    name="products_total_purchase_value",

    description="Total purchase value for all products",

    type="double",

    events=[

        Event(

            vendor="com.snowplowanalytics.snowplow.ecommerce",

            name="snowplow_ecommerce_action",

            version="1-0-2",

        )

    ],

    aggregation="sum",

    criteria=Criteria(

        all=[

            Criterion.eq(

                EventProperty(

                    vendor="com.snowplowanalytics.snowplow.ecommerce",

                    name="snowplow_ecommerce_action",

                    major_version=1,

                    path="type"

                ),

                "transaction"

            )

        ]

    ),

    property=EntityProperty(

        vendor="com.snowplowanalytics.snowplow.ecommerce",

        name="product",

        major_version=1,

        path="price",

    ),

    default_value=0

)
```

---

# Define attributes in Signals
> Define attributes within attribute groups to calculate behavioral data from real-time streams or your warehouse, then test and publish the configuration to Signals.
> Source: https://docs.snowplow.io/docs/signals/attributes/

[Attributes](/docs/signals/concepts/#attribute-groups) are the behavioral facts you want Signals to calculate, such as a user's page view count or lifetime value.

Attributes are defined within [attribute groups](/docs/signals/attributes/attribute-groups/). Each group specifies an [attribute key](/docs/signals/attributes/attribute-keys/) (the identifier to calculate against), the [attributes](/docs/signals/attributes/attributes/) to compute, and a data source — either a real-time event stream or a [warehouse table](/docs/signals/attributes/warehouse-config/).

This section covers the definition workflow:

- [Attribute groups](/docs/signals/attributes/attribute-groups/): create a group, choose a data source, and publish it
- [Attributes](/docs/signals/attributes/attributes/): configure the individual calculations within a group
- [Attribute keys](/docs/signals/attributes/attribute-keys/): use built-in identifiers or define custom ones
- [Warehouse configuration](/docs/signals/attributes/warehouse-config/): sync pre-calculated warehouse tables to Signals

To consume the calculated values in your applications, see [Use in your application](/docs/signals/applications/).

## Ways to define attributes

Each page in this section documents two approaches side by side, using tabs:

- **[Snowplow Console](https://console.snowplowanalytics.com)**: a UI for defining and managing attribute configurations. Navigate to the **Signals** section, and use the **Attribute groups**, **Attribute keys**, and **Services** pages to create and manage your configuration.
- **[Signals Python SDK](https://pypi.org/project/snowplow-signals/)**: define and publish attribute configurations programmatically from a notebook or script. Start by [connecting to Signals](/docs/signals/connection/).

You can also use the [Signals API](/docs/signals/connection/#signals-api) to interact with Signals directly over HTTP.

![Signals section of the Console navigation sidebar showing Overview, Attribute groups, Services, Attribute keys, and Interventions as menu items](/assets/images/console-navbar-11b65d884e2ac3450f8248e862b354c9.png)

## Configuration workflow

Attribute groups move through the same lifecycle whether you manage them in Console or with the Python SDK:

1. **Define**: [create the attribute group](/docs/signals/attributes/attribute-groups/#define-the-attribute-group), choosing its data source, attribute key, and attributes. In Console, new attribute groups are saved as drafts, so you can edit them freely before publishing.
2. **Test**: [preview the attribute values](/docs/signals/attributes/attribute-groups/#test-attribute-definitions) the group would produce, calculated from recent events in your warehouse.
3. **Publish**: [apply the configuration](/docs/signals/attributes/attribute-groups/#publish-attribute-groups) to your Signals infrastructure. Signals starts calculating attributes, or syncing warehouse tables, and populating the Profiles Store. Definitions only take effect once published: values are calculated from this point onwards, unless you've configured a backfill.
4. **Update**: editing a published attribute group creates a new [version](/docs/signals/attributes/attribute-groups/#versioning), leaving the published version unchanged. This means you can iterate on definitions without breaking downstream processes, then migrate consumers by updating your [service](/docs/signals/applications/services/) definitions.
5. **Unpublish**: [stop calculation](/docs/signals/attributes/attribute-groups/#delete-an-attribute-group) for a specific version without losing its definition. Existing values remain in the Profiles Store, and you can republish later.
6. **Delete**: permanently remove all versions of the group, along with its calculated values in the Profiles Store. Attribute groups must be unpublished before they can be deleted.

[Interventions](/docs/signals/interventions/) follow the same lifecycle. [Services](/docs/signals/applications/services/) are simpler: they're published automatically as soon as they're created, and aren't versioned.

Each page in this section, along with the services and interventions pages, documents the Console and Python SDK specifics for these steps.

---

# Sync warehouse tables to Signals
> Connect Signals to existing warehouse tables to sync pre-calculated attributes. Configure table details, timestamp fields, and field mappings using the Console or the Python SDK.
> Source: https://docs.snowplow.io/docs/signals/attributes/warehouse-config/

To sync existing, pre-calculated attributes from your warehouse to Signals, configure an attribute group with a warehouse source. No additional modeling is required — the batch engine reads rows from your warehouse table at a fixed interval and sends them to the Profiles Store.

> **Tip:** If your source data comes from Snowplow events, consider using a stream attribute group with the backfill option enabled instead. It gives you real-time updates from your event stream alongside historical backfill, without needing to maintain a separate warehouse table. Attributes that are fetched from a warehouse are best suited for pre-calculated values from non-Snowplow sources, or tables that already exist independently of your event pipeline.

## Provide source table details

**Console:**

When creating an attribute group, select **Warehouse** as the data source.

![Create attribute group form showing where to select the Warehouse data source in Snowplow Console](/assets/images/warehouse-9eec53b7ae9891c9c33395b79ea610c1.png)

Then provide the warehouse and table details, and select which fields you want to send to Signals.

![Warehouse source configuration showing warehouse table and field mapping options](/assets/images/attribute-group-warehouse-fields-645921d8d99b1a8c4cd7aec3ac157b2a.png)

**Python SDK:**

Configure which table to sync by specifying a `BatchSource` object.

```python
from snowplow_signals import BatchSource



data_source = BatchSource(

    name="ecommerce_transaction_interactions_source",

    database="SNOWPLOW_DEV1",

    schema="SIGNALS",

    table="SNOWPLOW_ECOMMERCE_TRANSACTION_INTERACTIONS_FEATURES",

    timestamp_field="UPDATED_AT",

    owner="user@company.com",

)
```

The table below lists all available arguments for a `BatchSource`:

| Argument          | Description                                                         | Type     | Required? |
| ----------------- | ------------------------------------------------------------------- | -------- | --------- |
| `name`            | The name of the source                                              | `string` | ✅         |
| `description`     | A description of the source                                         | `string` | ❌         |
| `database`        | The database where the attributes are stored                        | `string` | ✅         |
| `schema`          | The schema for the table of interest                                | `string` | ✅         |
| `table`           | The table where the attributes are stored                           | `string` | ✅         |
| `timestamp_field` | Primary timestamp of the attribute value, indicating data freshness | `string` | ❌         |
| `owner`           | The owner of the source                                             | `string` | ❌         |

The batch engine only sends rows with a newer timestamp to the Profiles Store, based on the `timestamp_field`. If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the greatest `timestamp_field` value.

***

## Define which fields to sync

**Console:**

Select the fields (columns) from your warehouse table that you want to sync to the Profiles Store. You must also specify a backfill start date — the earliest date from which Signals should read rows on the first sync.

The [attribute key](/docs/signals/attributes/attribute-keys/) for the group must correspond to a column in the table that contains the key values.

**Python SDK:**

Use `ExternalBatchAttributeGroup` to define which source table and fields to use. Instead of `attributes`, this class uses `fields` — abstractions over the warehouse columns.

You must set `backfill_since_tstamp` to tell the batch engine the earliest timestamp from which to read rows on the first sync. Without this, no historical data will be loaded.

The attribute key must correspond to a column in the source table. The example below uses the built-in `domain_userid` key, so the table must have a `domain_userid` column. To key on a different column, [define a custom attribute key](/docs/signals/attributes/attribute-keys/#custom-attribute-keys) with `external_column` set to the column name.

```python
from datetime import datetime, timezone

from snowplow_signals import ExternalBatchAttributeGroup, domain_userid, Field



attribute_group = ExternalBatchAttributeGroup(

    name="ecommerce_transaction_interactions_attributes",

    version=1,

    attribute_key=domain_userid,

    owner="user@company.com",

    batch_source=data_source,

    backfill_since_tstamp=datetime(2026, 6, 1, tzinfo=timezone.utc),

    fields=[

        Field(name="TOTAL_TRANSACTIONS", type="int32"),

        Field(name="TOTAL_REVENUE", type="int32"),

        Field(name="AVG_TRANSACTION_REVENUE", type="int32"),

    ],

)
```

The table below lists all available arguments for `ExternalBatchAttributeGroup`:

| Argument                | Description                                                                                                                          | Type           | Required? |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------- | --------- |
| `name`                  | The name of the attribute group                                                                                                      | `string`       | ✅         |
| `version`               | The version of the attribute group                                                                                                   | `int`          | ✅         |
| `attribute_key`         | The key used to identify profiles. Its name must match a column in the source table                                                  | `AttributeKey` | ✅         |
| `batch_source`          | The `BatchSource` defining the warehouse table to sync from                                                                          | `BatchSource`  | ✅         |
| `backfill_since_tstamp` | The earliest timestamp from which to read rows on the first sync. Accepts tz-aware or naive UTC `datetime`; tz-aware is recommended. | `datetime`     | ✅         |
| `fields`                | The list of `Field` objects defining which columns to sync                                                                           | `list`         | ✅         |
| `description`           | A description of the attribute group                                                                                                 | `string`       | ❌         |
| `owner`                 | The owner of the attribute group                                                                                                     | `string`       | ❌         |

The table below lists all available arguments for a `Field`:

| Argument      | Description                | Type                                                                                                                                                                                                               | Required? |
| ------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------- |
| `name`        | The name of the field      | `string`                                                                                                                                                                                                           | ✅         |
| `description` | A description of the field | `string`                                                                                                                                                                                                           | ❌         |
| `type`        | The type of the field      | one of: `bytes`, `string`, `int32`, `int64`, `double`, `float`, `bool`, `unix_timestamp`, `bytes_list`, `string_list`, `int32_list`, `int64_list`, `double_list`, `float_list`, `bool_list`, `unix_timestamp_list` | ✅         |

***

---

# Signals fundamentals
> Signals introduces attribute groups for defining behavioral data, services for consuming attributes, and interventions for triggering actions. Learn about data sources, attribute keys, and the Profiles Store.
> Source: https://docs.snowplow.io/docs/signals/concepts/

Signals introduces a new set of data governance concepts to Snowplow. As with schemas for Snowplow event data, Signals components are strictly defined, structured, and versioned.

Signals has three main configurable components:

- **Attribute groups**, for defining and calculating attributes
- **Services**, for consuming calculated attributes in your applications
- **Interventions**, for consuming calculated attributes and triggering actions in your applications

**Attribute groups** are where you define the behavioral data you want to calculate. Each attribute group contains multiple **attributes** - the specific facts about user behavior you want to measure or take action on - along with the configuration that defines how to calculate them, and from what data. Attributes can only be defined within attribute groups; they are effectively properties of the attribute group.

To use attributes to trigger actions such as in-app messages, discounts, or personalized journeys, use services or interventions.

**Services** provide a stable interface layer between your calculated attributes and your applications. Each service can contain multiple attribute groups, pinned to specific versions. You'd build the logic within your application for how to use the retrieved attributes. **Interventions** are a separate abstraction for defining when to trigger actions in your application.

![Signals component flow showing Snowplow events and batch sources feeding attribute groups, which are calculated into the Profiles Store and consumed by applications through services and interventions](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJiYWNrZ3JvdW5kOiAjZmZmZmZmOyBiYWNrZ3JvdW5kLWNvbG9yOiBsaWdodC1kYXJrKCNmZmZmZmYsIHZhcigtLWdlLWRhcmstY29sb3IsICMxMjEyMTIpKTsgY29sb3Itc2NoZW1lOiBsaWdodCBkYXJrOyIgdmVyc2lvbj0iMS4xIiB2aWV3Qm94PSIwIDAgMTY4MCAyODAiIGZvbnQtZmFtaWx5PSJJbnRlciwgLWFwcGxlLXN5c3RlbSwgJ1NlZ29lIFVJJywgSGVsdmV0aWNhLCBBcmlhbCwgc2Fucy1zZXJpZiI+CiAgPGRlZnM+CiAgICA8bWFya2VyIGlkPSJhcnJvdy1wdXJwbGUiIHZpZXdCb3g9IjAgMCAxMCAxMCIgcmVmWD0iOSIgcmVmWT0iNSIgbWFya2VyV2lkdGg9IjYiIG1hcmtlckhlaWdodD0iNiIgb3JpZW50PSJhdXRvLXN0YXJ0LXJldmVyc2UiPgogICAgICA8cGF0aCBkPSJNIDAgMSBMIDkgNSBMIDAgOSB6IiBmaWxsPSIjNjYzOGI4Ii8+CiAgICA8L21hcmtlcj4KICAgIDxtYXJrZXIgaWQ9ImFycm93LWdyZWVuIiB2aWV3Qm94PSIwIDAgMTAgMTAiIHJlZlg9IjkiIHJlZlk9IjUiIG1hcmtlcldpZHRoPSI2IiBtYXJrZXJIZWlnaHQ9IjYiIG9yaWVudD0iYXV0by1zdGFydC1yZXZlcnNlIj4KICAgICAgPHBhdGggZD0iTSAwIDEgTCA5IDUgTCAwIDkgeiIgZmlsbD0iIzFkYjU4ZiIvPgogICAgPC9tYXJrZXI+CiAgPC9kZWZzPgoKICA8IS0tIGNvbm5lY3RvcnMgLS0+CiAgPGcgZmlsbD0ibm9uZSIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtZGFzaGFycmF5PSI2IDUiPgogICAgPHBhdGggZD0iTSAyMzAgNjUgSCAzMzAgViAxMjUgSCA0MzAiIHN0cm9rZT0iIzY2MzhiOCIgbWFya2VyLWVuZD0idXJsKCNhcnJvdy1wdXJwbGUpIi8+CiAgICA8cGF0aCBkPSJNIDIzMCAxODUgSCAzMzAgViAxMzAiIHN0cm9rZT0iIzY2MzhiOCIvPgogICAgPHBhdGggZD0iTSA2NzAgMTI1IEggODUwIiBzdHJva2U9IiM2NjM4YjgiIG1hcmtlci1lbmQ9InVybCgjYXJyb3ctcHVycGxlKSIvPgogICAgPHBhdGggZD0iTSAxMDgwIDEyNSBIIDExMzAgViA2NSBIIDExNjUiIHN0cm9rZT0iIzY2MzhiOCIgbWFya2VyLWVuZD0idXJsKCNhcnJvdy1wdXJwbGUpIi8+CiAgICA8cGF0aCBkPSJNIDExMzAgMTI1IFYgMTg1IEggMTE2NSIgc3Ryb2tlPSIjNjYzOGI4IiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93LXB1cnBsZSkiLz4KICAgIDxwYXRoIGQ9Ik0gMTM4MCA2NSBIIDE0MjAgViAxMDUgSCAxNDQ1IiBzdHJva2U9IiM2NjM4YjgiIG1hcmtlci1lbmQ9InVybCgjYXJyb3ctcHVycGxlKSIvPgogICAgPHBhdGggZD0iTSAxMzgwIDE4NSBIIDE0MjAgViAxNDUgSCAxNDQ1IiBzdHJva2U9IiMxZGI1OGYiIG1hcmtlci1lbmQ9InVybCgjYXJyb3ctZ3JlZW4pIi8+CiAgPC9nPgoKICA8IS0tIGNvbm5lY3RvciBvcmlnaW4gZG90cyAtLT4KICA8ZyBmaWxsPSIjNjYzOGI4Ij4KICAgIDxjaXJjbGUgY3g9IjIzMCIgY3k9IjY1IiByPSI1Ii8+CiAgICA8Y2lyY2xlIGN4PSIyMzAiIGN5PSIxODUiIHI9IjUiLz4KICAgIDxjaXJjbGUgY3g9IjY3MCIgY3k9IjEyNSIgcj0iNSIvPgogICAgPGNpcmNsZSBjeD0iMTA4MCIgY3k9IjEyNSIgcj0iNSIvPgogICAgPGNpcmNsZSBjeD0iMTM4MCIgY3k9IjY1IiByPSI1Ii8+CiAgPC9nPgogIDxjaXJjbGUgY3g9IjEzODAiIGN5PSIxODUiIHI9IjUiIGZpbGw9IiMxZGI1OGYiLz4KCiAgPCEtLSBub2RlcyAtLT4KICA8ZyBzdHJva2U9IiM2NjM4YjgiIHN0cm9rZS13aWR0aD0iMS41Ij4KICAgIDxyZWN0IHg9IjIwIiB5PSIzMCIgd2lkdGg9IjIxMCIgaGVpZ2h0PSI3MCIgcng9IjM1IiBmaWxsPSIjZmZmZmZmIi8+CiAgICA8cmVjdCB4PSIxNCIgeT0iMTYyIiB3aWR0aD0iMjEwIiBoZWlnaHQ9IjcwIiByeD0iMzUiIGZpbGw9IiNmZmZmZmYiLz4KICAgIDxyZWN0IHg9IjE3IiB5PSIxNTYiIHdpZHRoPSIyMTAiIGhlaWdodD0iNzAiIHJ4PSIzNSIgZmlsbD0iI2ZmZmZmZiIvPgogICAgPHJlY3QgeD0iMjAiIHk9IjE1MCIgd2lkdGg9IjIxMCIgaGVpZ2h0PSI3MCIgcng9IjM1IiBmaWxsPSIjZmZmZmZmIi8+CiAgICA8cmVjdCB4PSI0NDAiIHk9IjkwIiB3aWR0aD0iMjMwIiBoZWlnaHQ9IjcwIiByeD0iMzUiIGZpbGw9IiNmZmZmZmYiLz4KICAgIDxyZWN0IHg9Ijg2MCIgeT0iNTAiIHdpZHRoPSIyMjAiIGhlaWdodD0iMTUwIiByeD0iMTgiIGZpbGw9IiNmNGVlZjkiLz4KICAgIDxyZWN0IHg9IjExNzUiIHk9IjMwIiB3aWR0aD0iMjA1IiBoZWlnaHQ9IjcwIiByeD0iMzUiIGZpbGw9IiNmZmZmZmYiLz4KICAgIDxyZWN0IHg9IjExNzUiIHk9IjE1MCIgd2lkdGg9IjIwNSIgaGVpZ2h0PSI3MCIgcng9IjM1IiBmaWxsPSIjZmZmZmZmIi8+CiAgICA8cmVjdCB4PSIxNDU1IiB5PSI1MCIgd2lkdGg9IjIwNSIgaGVpZ2h0PSIxNTAiIHJ4PSIxOCIgZmlsbD0iI2ZmZmZmZiIvPgogIDwvZz4KCiAgPCEtLSBQcm9maWxlcyBTdG9yZSBwZXJzb24gaWNvbiAtLT4KICA8ZyBzdHJva2U9IiM2NjM4YjgiIHN0cm9rZS13aWR0aD0iMiIgZmlsbD0ibm9uZSI+CiAgICA8Y2lyY2xlIGN4PSI5NzAiIGN5PSI5OCIgcj0iMjIiLz4KICAgIDxjaXJjbGUgY3g9Ijk3MCIgY3k9IjkxIiByPSI2LjUiLz4KICAgIDxwYXRoIGQ9Ik0gOTU4IDExMCBDIDk2MSAxMDEgOTc5IDEwMSA5ODIgMTEwIi8+CiAgPC9nPgoKICA8IS0tIEFwcGxpY2F0aW9uIHdpbmRvdyBpY29uIC0tPgogIDxnIHN0cm9rZT0iIzY2MzhiOCIgc3Ryb2tlLXdpZHRoPSIyIiBmaWxsPSJub25lIj4KICAgIDxjaXJjbGUgY3g9IjE1NTciIGN5PSI5OCIgcj0iMjIiLz4KICAgIDxyZWN0IHg9IjE1NDYiIHk9Ijg5IiB3aWR0aD0iMjIiIGhlaWdodD0iMTgiIHJ4PSIzIi8+CiAgICA8bGluZSB4MT0iMTU0NiIgeTE9Ijk1IiB4Mj0iMTU2OCIgeTI9Ijk1Ii8+CiAgPC9nPgoKICA8IS0tIGxhYmVscyAtLT4KICA8ZyBmaWxsPSIjMWYyOTM3IiBmb250LXNpemU9IjE3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj4KICAgIDx0ZXh0IHg9IjEyNSIgeT0iNzEiPlNub3dwbG93IGV2ZW50czwvdGV4dD4KICAgIDx0ZXh0IHg9IjEyNSIgeT0iMTkxIj5CYXRjaCBzb3VyY2VzPC90ZXh0PgogICAgPHRleHQgeD0iNTU1IiB5PSIxMzEiPkF0dHJpYnV0ZSBncm91cHM8L3RleHQ+CiAgICA8dGV4dCB4PSI5NzAiIHk9IjE2OCI+UHJvZmlsZXMgU3RvcmU8L3RleHQ+CiAgICA8dGV4dCB4PSIxMjc3IiB5PSI3MSI+U2VydmljZXM8L3RleHQ+CiAgICA8dGV4dCB4PSIxMjc3IiB5PSIxOTEiPkludGVydmVudGlvbnM8L3RleHQ+CiAgICA8dGV4dCB4PSIxNTU3IiB5PSIxNjgiPkFwcGxpY2F0aW9uPC90ZXh0PgogIDwvZz4KPC9zdmc+Cg==)

## Attribute groups

Attribute groups are where you define the data you want to calculate. Each attribute group is a versioned collection that specifies:

- The **attributes** to calculate - the specific behavioral facts about users
- The **data source** - whether to calculate from the real-time stream, or in batch from the warehouse
- The **attribute key** that provides the analytical context
- Other metadata such as description or owner

### Types of attribute

Attributes describe what kind of calculation to perform, and what event data to evaluate. They can only exist within attribute groups.

Attributes can be categorized into four main types, depending on the type of user behavior you want to understand:

| Type                | Description                                         | Example                              |
| ------------------- | --------------------------------------------------- | ------------------------------------ |
| Period-based        | Actions that happened within a configured period    | `products_added_to_cart_last_10_min` |
| No-period aggregate | Calculated over all available data (subject to TTL) | `total_product_price_clv`            |
| First touch         | The first event or property that happened           | `first_mkt_source`                   |
| Last touch          | The most recent event or property that happened     | `last_device_class`                  |

For period-based attributes, the period is configured on each attribute. Data retention is controlled by the TTL configured on the attribute group.

Signals includes a range of different aggregations for calculating attributes, including `mean`, `counter`, or `unique_list`. See the full list in the [attribute configuration](/docs/signals/attributes/attributes/) page.

### Attribute keys

An attribute key is an identifier that provides the analytical context for all attribute calculations within a group. For stream attribute groups, the identifier can be any property of a Snowplow event: an atomic field such as `domain_userid`, a property within a [self-describing event](/docs/fundamentals/events/#self-describing-events), or a property within an [entity](/docs/fundamentals/entities/). For warehouse attribute groups, the identifier is a column in your warehouse table.

To demonstrate the necessity of attribute keys, consider the attribute `num_views_in_last_10_min`. It represents a count of page view events, with a 10 minute time limit. This table lists some possible meanings of the attribute, based on the attribute key configured for its attribute group:

| Attribute                  | Attribute key      | Description                                                                         |
| -------------------------- | ------------------ | ----------------------------------------------------------------------------------- |
| `num_views_in_last_10_min` | User               | How many pages a user has viewed within the last 10 minutes                         |
| `num_views_in_last_10_min` | Page               | How many page views a page has received within the last 10 minutes                  |
| `num_views_in_last_10_min` | Product            | How many times a product has been viewed within the last 10 minutes                 |
| `num_views_in_last_10_min` | App                | How many page views occurred within an app in the last 10 minutes                   |
| `num_views_in_last_10_min` | Device             | How many page views came from a specific device in the last 10 minutes              |
| `num_views_in_last_10_min` | Marketing campaign | How many page views were generated by a campaign in the last 10 minutes             |
| `num_views_in_last_10_min` | Geographic region  | How many page views came from users in one region within the last 10 minutes        |
| `num_views_in_last_10_min` | Customer segment   | How many page views were generated by users in a segment within the last 10 minutes |

Each of these is likely to have a different calculated value.

You can [define your own attribute keys](/docs/signals/attributes/attribute-groups/#attribute-keys), or use the built-in ones. Signals comes with predefined attribute keys for users and sessions. Their identifiers are from the out-of-the-box atomic [user-related fields](/docs/fundamentals/canonical-event/#user-fields) in all Snowplow events.

| Attribute key      | Type     |
| ------------------ | -------- |
| `user_id`          | `string` |
| `domain_userid`    | `uuid`   |
| `network_userid`   | `uuid`   |
| `domain_sessionid` | `uuid`   |

## Data sources

Whether to compute attributes in real-time from the event stream or to sync pre-calculated values from your warehouse is an important decision. Broadly, you might use:

- **Stream** for real-time use cases, such as tracking the latest product a user viewed, or the number of page views in a session. You can optionally enable backfill to enrich stream attributes with historical data from your warehouse, such as a user's purchase history.
- **Warehouse** to bring in modeled data from your warehouse, such as transactional data or CRM attributes

This table summarizes the options for different types of processing:

| Feature                                           | Stream                                                            | Stream with backfill                                                    | Warehouse                 |
| ------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------- |
| Real-time calculation                             | ✅                                                                 | ✅                                                                       | ❌                         |
| Period-based calculations                         | ✅                                                                 | ✅                                                                       | ❌                         |
| Calculations over all available data (within TTL) | ✅ from the point at which the attribute was defined               | ✅ from the configured backfill start date                               | ❌                         |
| Historical data                                   | ❌ attributes are only calculated from the moment they are defined | ✅ backfilled from the `atomic` events table up to the publish timestamp | ✅ any pre-calculated data |
| Non-Snowplow data                                 | ❌                                                                 | ❌                                                                       | ✅                         |
| Warehouse connection required                     | ❌                                                                 | ✅                                                                       | ✅                         |

### Stream source

When Signals is deployed in your pipeline, the event stream is read by the streaming engine. All tracked events are inspected.

Real-time stream flow:

1. Behavioral data event is received by [Collector](/docs/pipeline/collector/)

2. Event is enriched by [Enrich](/docs/pipeline/enrichments/)

3. The Signals stream engine reads from the events stream

4. The stream engine checks if attributes are defined for the event

5. Are there any attributes to calculate?

   - No: nothing happens, and the process ends
   - Yes: processing continues

6. Signals evaluates and computes the attributes

7. Updated attributes are pushed to the Profiles Store

#### Backfill

Stream attribute groups by default only calculate attributes from the moment they are published. If you would like to pre-populate your attributes with values from your warehouse, you can optionally enable backfill when creating the group. Backfilling stream attributes requires a warehouse connection.

When backfill is enabled, you select a start date. On publish, Signals uses your Snowplow `atomic` events table to calculate attribute values for all events from that date up to the publish timestamp. The streaming engine starts immediately and processes all new events in real time. Backfill operates within this defined period and can take some time depending on data volume. Once backfill completes, only the streaming engine remains active.

You can monitor progress in Snowplow Console on the attribute group's details page. While historical values are syncing, a status bar labeled `Backfill in progress` is shown. This status remains visible until the backfill period has completed.

![Backfill progress status bar shown for stream with backfill attribute groups](/assets/images/backfill_progress-743f0b5dd6a28702ad6a27a26fc3432d.png)

### Warehouse source

Use a warehouse source to sync tables of existing, pre-calculated values to the Profiles Store. The warehouse tables can be any data. For example, you may want to include transactional data or CRM attributes in your Signals use case.

## Services

Services are where you define how to use the calculated attributes in your applications.

Rather than connecting applications directly to attribute groups, services allow you to consume specific attribute group versions. This provides a consistent set of consumable attributes. We strongly recommend using services in production applications.

By using services you can:

- Iterate on attribute definitions without worrying about breaking downstream processes
- Migrate to new attribute group versions by updating the service definition, without having to update the application code

## Interventions

Interventions are opportunities to take actions to improve user outcomes. They're automated triggers fired by changes in attribute values, or by your own applications.

They can be thought of as "if-then" rules that run in real time. For example:

- `If` a user has viewed five products in the last ten minutes
- `Then` automatically send that user a personalized offer

In practice, this rule could work like this:

1. User views product pages
2. The application tracks the user behavior and sends events
3. Signals evaluates the events and updates attribute values
4. Signals checks whether the new attribute values meet any intervention conditions
5. The attribute `product_view_count_last_10_min` has just been updated to `5`, matching the intervention's trigger criteria
6. The application receives the subscribed intervention payload
7. The application sends a personalized offer to the user

Interventions are usually relevant only within a certain moment. Therefore, an intervention is sent only the first time the criteria are met.

Standard rule-based interventions can have multiple conditions, or trigger criteria, based on the values of different attributes.

### Targeting

Interventions are targeted based on attribute keys, which determine both the scope and specificity of when they're delivered.

For individual-level targeting, use user-specific attribute keys. For example, use `domain_userid` to target individual users, or `domain_sessionid` to target users during specific sessions, when session-level conditions are met.

For broadcast-level targeting, use attribute keys related to the application context. For example, you could use `campaign_id` to target all users who arrived from a specific marketing campaign.

Interventions can have multiple attribute keys. By default, the intervention will target the attribute keys associated with their criteria attributes. See [Define interventions](/docs/signals/interventions/#target-and-payload) for guidance on choosing custom targets.

### Subscribing

To receive and take action on interventions, you'll need to [subscribe](/docs/signals/applications/subscribe/) to them within your application, and define the logic for how the application should react. Once subscribed, all triggered interventions are streamed to the consumer application.

Subscription is by attribute key, not by intervention: a subscription using a specific attribute key ID, for example a `domain_userid` ID for the current user, will receive all triggered interventions for that ID. See the [delivery example](/docs/signals/applications/subscribe/#delivery-example) for how targeting, subscription, and the send-once rule work together.

### Types of intervention

Signals has two types of interventions: rule-based or direct.

**Rule-based** interventions are the default, and are:

- Based on attribute values
- Defined in advance via Snowplow Console, Python SDK, or API
- Triggered automatically when their criteria are met

Use cases include:

- Real-time personalization
- Agentic chatbots
- Dynamic pricing

**Direct** interventions are:

- Not based on attribute values or rules
- Configurable only via the Python SDK or API
- Sent directly on pushing an intervention configuration to Signals

Use cases for direct interventions include:

- Emergency notifications, e.g. system outages
- Real-time business decisions, e.g. breaking news
- Manual campaign targeting
- Sensitive communications requiring authentication or authorization

## Profiles Store

The Profiles Store is a database where Signals saves all your calculated attribute values. When Signals calculates attributes from your events or warehouse data, or syncs pre-calculated data, it stores them here organized by attribute group. Your applications retrieve these stored values using the Signals SDKs or API.

The Profiles Store keeps track of current attribute values, and automatically removes old data based on the TTL settings you configure for each attribute group. It acts as the central source of truth for your Signals deployment.

---

# Connect to Snowplow Signals
> Connect to Signals using the Python SDK, Node.js SDK, or REST API with your deployment credentials, to manage configurations or consume calculated values.
> Source: https://docs.snowplow.io/docs/signals/connection/

You'll connect to Signals for one of two reasons:

- **To define and manage configurations**: create attribute groups, services, and interventions, and publish them to Signals. This is typically done by a data team, using Snowplow Console, the Python SDK, or the API.
- **To consume calculated values in your application**: retrieve attributes on demand, or subscribe to interventions. This is typically done in application code, using the SDKs, browser plugin, or API.

This table shows which interface supports which task:

| Interface                                                 | Define configurations | Retrieve attributes | Subscribe to interventions |
| --------------------------------------------------------- | --------------------- | ------------------- | -------------------------- |
| [Snowplow Console](https://console.snowplowanalytics.com) | ✅                     | ❌                   | ❌                          |
| [Python SDK](#signals-python-sdk)                         | ✅                     | ✅                   | ✅                          |
| [Node.js SDK](#signals-nodejs-sdk)                        | ❌                     | ✅                   | ❌                          |
| [Browser plugin](/docs/signals/applications/subscribe/)   | ❌                     | ❌                   | ✅                          |
| [Signals API](#signals-api)                               | ✅                     | ✅                   | ✅                          |

Console doesn't require any connection setup: log in and navigate to the **Signals** section. If Signals isn't enabled for your organization yet, see [Set up Signals](/docs/signals/setup/).

For the SDKs, plugin, and API, you'll need [connection credentials](#connection-credentials).

## Connection credentials

To connect to Signals using the Signals SDKs or API, you will need four values. Use the Console **Overview** page to access them.

| Value           | Description                             | Where to get it                                          | Format                                             |
| --------------- | --------------------------------------- | -------------------------------------------------------- | -------------------------------------------------- |
| Signals API URL | The API URL for your Signals deployment | Console > **Signals** > **Overview**                     | `https://{{123abc}}.signals.snowplowanalytics.com` |
| API key         | A Snowplow API key                      | [Generated by you](/docs/account-management/) in Console | UUID                                               |
| API key ID      | A Snowplow API key ID                   | [Generated by you](/docs/account-management/) in Console | UUID                                               |
| Organization ID | Your Snowplow organization ID           | Console > **Signals** > **Overview**                     | UUID                                               |

Add these four tokens to your environment secrets.

## Signals Python SDK

Use the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) to define attribute groups, services, and interventions via code. You can also retrieve calculated attributes and subscribe to interventions using this SDK.

To use the Python SDK, first choose where to store your Signals configurations. We recommend creating a new repository. The easiest way to use the SDK is within a Jupyter notebook.

Install the SDK into your environment:

```bash
pip install snowplow-signals
```

Once installed, you can start to define configuration components. To test or publish your configuration, or retrieve calculated attributes or interventions from the Profiles Store, you'll need to connect to your Signals deployment.

Create a `Signals` object by passing in the required values:

```python
from snowplow_signals import Signals



sp_signals = Signals(

    api_url=SIGNALS_DEPLOYED_URL,

    api_key=CONSOLE_API_KEY,

    api_key_id=CONSOLE_API_KEY_ID,

    org_id=ORG_ID,

)
```

The created `Signals` object has the following methods:

| Method                   | Description                                                                 |
| ------------------------ | --------------------------------------------------------------------------- |
| `publish`                | Registers the provided objects with Signals                                 |
| `unpublish`              | Unpublishes objects from Signals                                            |
| `delete`                 | Fully deletes objects from Signals (must unpublish first)                   |
| `test`                   | Tests an attribute group against the atomic events table                    |
| `get_service_attributes` | Retrieves attributes for a specific service from the Profiles Store         |
| `get_group_attributes`   | Retrieves attributes for a specific attribute group from the Profiles Store |
| `get_attribute_group`    | Retrieves an attribute group from the Profiles Store                        |
| `push_intervention`      | Push an intervention to subscribers for a set of attribute keys             |
| `pull_interventions`     | Open a streaming subscription of interventions for a set of attribute keys  |

To define configurations programmatically, check out the [attribute groups](/docs/signals/attributes/attribute-groups/), [services](/docs/signals/applications/services/), and [interventions](/docs/signals/interventions/) pages. The [configuration workflow](/docs/signals/attributes/#configuration-workflow) describes how publishing, versioning, unpublishing, and deletion fit together.

To consume calculated values in your application, see [retrieve attributes](/docs/signals/applications/retrieve-attributes/) and [subscribe to interventions](/docs/signals/applications/subscribe/).

## Signals Node.js SDK

Use the [Node.js SDK](https://www.npmjs.com/package/@snowplow/signals-node) to retrieve calculated attributes.

Install the SDK into your application:

```bash
npm i @snowplow/signals-node

# or

yarn add @snowplow/signals-node

# or

pnpm i @snowplow/signals-node
```

Create a `Signals` object by passing in the required values:

```typescript
import { Signals } from '@snowplow/signals-node';



const signals = new Signals({

  baseUrl: SIGNALS_DEPLOYED_URL,

  apiKey: CONSOLE_API_KEY,

  apiKeyId: CONSOLE_API_KEY_ID,

  organizationId: ORG_ID,

});
```

The created `Signals` object has the following methods:

| Method                      | Description                                                                 |
| --------------------------- | --------------------------------------------------------------------------- |
| `getServiceAttributes`      | Retrieves attributes for a specific service from the Profiles Store         |
| `getGroupAttributes`        | Retrieves attributes for a specific attribute group from the Profiles Store |
| `getBatchServiceAttributes` | Retrieves attributes for multiple identifiers from a service                |

See [retrieve attributes](/docs/signals/applications/retrieve-attributes/) for usage examples.

## Signals API

The Signals API allows you to directly configure and retrieve attributes and interventions. To access the full Swagger API documentation for your Signals deployment, use your Signals API URL followed by `/docs/`:

```bash
{{API_URL}}/docs/
```

Your API documentation is linked in Console on the **Overview** page, under **Configuration details**.

### Authenticate to the Signals API

The SDKs handle authentication for you. When you call the Signals API directly, each request must include a Bearer token in the `Authorization` header.

To obtain this token, exchange your Snowplow API key and key ID for a temporary access token using the Console Credentials API. The token is a JWT valid for 24 hours. See [Account management](/docs/account-management/#obtain-an-access-token) for the full process, including how to create an API key.

First, exchange your API key and key ID for an access token:

```bash
curl \

  --header 'X-API-Key-ID: <API_KEY_ID>' \

  --header 'X-API-Key: <API_KEY>' \

  https://console.snowplowanalytics.com/api/msc/v1/organizations/<ORGANIZATION_ID>/credentials/v3/token
```

This returns a JWT:

```json
{ "accessToken": "<JWT>" }
```

Then use the JWT as a Bearer token in your Signals API requests:

```bash
curl \

  --header 'Authorization: Bearer <JWT>' \

  {{API_URL}}/api/v1/registry/interventions
```

---

# Get started with Snowplow Signals
> Learn how to use Snowplow Signals to compute user attributes from event data, retrieve them in real time for personalization and agentic applications, and trigger automated actions based on user behavior.
> Source: https://docs.snowplow.io/docs/signals/get-started/

Snowplow Signals computes user attributes from your data. You can retrieve these attributes in your applications to take real-time actions. Signals can also send automated triggers to your applications based on calculated attributes.

Use Signals to:

- Provide enriched user context to chatbots and other agentic applications, in real time
- Deliver personalized recommendations, dynamic pricing, and adaptive UIs based on current behavior
- Trigger actions automatically when users meet specific criteria

By default, Signals calculates attributes from your real-time Snowplow event stream, but you can also sync pre-calculated values from your warehouse.

If you don't have a Snowplow account yet, sign up for a [Snowplow free trial](https://snowplow.io/get-started/snowplow-free-trial) to experience Signals and Snowplow Console.

## Architecture

You will need a [Snowplow pipeline](/docs/get-started/) to use Signals. Your Signals infrastructure is deployed into the same cloud as your pipeline.

> **Note:** Only Snowflake and BigQuery are supported currently. However, you can also use Signals without the warehouse functionality.

Signals consists of several new infrastructure components. When running Signals, your Snowplow pipeline will continue to process events as usual.

The core Signals components are:

- **Profiles Store**: stores calculated attributes and configurations
- Signals **SDKs** and **API**: allow you to manage and fetch attributes and interventions
- **Streaming engine**: computes attributes from Snowplow events in real time and sends them directly to the Profiles Store
- **Batch engine**: reads pre-calculated attribute values from your warehouse tables at a fixed interval and syncs them to the Profiles Store; also backfills historical data for stream attribute groups

![Snowplow Signals architecture diagram showing core components including Profiles Store, SDKs, streaming engine, and batch engine](/assets/images/overview-incl-batch-engine-8a9efdb4fc0d4788a8598e41e514a823.svg)

## Workflow

These are the high-level steps for using Signals, and the documentation in this section follows the same order:

1. Set up Signals
2. Plan your use case
3. Define and publish your configuration
4. Take action in your application

Check out the [quick start tutorial](/tutorials/signals-quickstart/start/) for a hands-on walkthrough.

### 1. Set up Signals

[Enable Signals](/docs/signals/setup/) in Snowplow Console. This is a one-time task that deploys the Signals infrastructure alongside your pipeline.

Connecting a warehouse is optional, but you'll need one to backfill stream attributes with historical data, sync pre-calculated warehouse tables, or test attribute definitions before publishing.

### 2. Plan your use case

Decide what changes in user behavior you're aiming for, and what your application will do about them. This planning will tell you:

- Which [attributes](/docs/signals/concepts/#attribute-groups) to calculate, and at what level — per user, per session, or against a custom [attribute key](/docs/signals/concepts/#attribute-keys)
- Whether to calculate them from your real-time event stream, sync pre-calculated values from your warehouse, or both — see [data sources](/docs/signals/concepts/#data-sources)
- Whether your application will pull attribute values on demand, react to [interventions](/docs/signals/concepts/#interventions) pushed by Signals, or both

The [fundamentals](/docs/signals/concepts/) page explains all of these components.

### 3. Define and publish your configuration

[Define attribute groups](/docs/signals/attributes/) and [interventions](/docs/signals/interventions/), using Snowplow Console or the [Signals Python SDK](/docs/signals/connection/#signals-python-sdk). Group your published attribute groups into [services](/docs/signals/applications/services/), so your applications consume a stable set of attributes.

Definitions only take effect once published: publishing is what makes Signals start calculating attributes and populating the Profiles Store. The [configuration workflow](/docs/signals/attributes/#configuration-workflow) describes the full lifecycle, including testing definitions before you publish.

### 4. Take action in your application

[Retrieve the calculated attributes](/docs/signals/applications/retrieve-attributes/) in your application using the Python SDK, Node.js SDK, or API, and use them to personalize the user experience.

To react to user behavior as it happens, [subscribe to interventions](/docs/signals/applications/subscribe/) using the Python SDK, the browser tracker plugin, or the API.

## Resources

Check out the Signals foundations tutorials:

- [Quick Start](/tutorials/signals-quickstart/start/) for defining stream attributes using the UI or the Python SDK

Follow the [real-time prospect scoring with ML](/tutorials/signals-ml-prospect-scoring/intro/) solution accelerator to explore using Signals with a machine learning use case.

---

# Define interventions to trigger real-time actions with Signals
> Create rule-based interventions with criteria and targeting to automatically trigger actions when attribute values change, or push direct interventions. Configure using Console UI, Python SDK, or API.
> Source: https://docs.snowplow.io/docs/signals/interventions/

[Interventions](/docs/signals/concepts/#interventions) are automated triggers that enable real-time actions based on user behavior. You'll need to define them, as described on this page, and [subscribe](/docs/signals/applications/subscribe/) to them in your application.

There are three methods for defining interventions in Signals:

- Snowplow Console UI
- [Signals Python SDK](https://pypi.org/project/snowplow-signals/)
- [Signals API](/docs/signals/connection/#signals-api)

Signals has two [types of interventions](/docs/signals/concepts/#types-of-intervention): rule-based interventions, which trigger automatically when their criteria are met, and [direct interventions](#direct-interventions), which you push manually using the SDK or API.

**Console:**

To create an intervention using the UI, go to **Signals** > **Interventions** in Snowplow Console and follow the instructions.

The first step is to specify:

- A unique name
- An optional description
- The email address of the primary owner or maintainer

![Create intervention form with name, description, and owner fields](/assets/images/intervention-create-92808b8ed9db1d48636405066ce3df83.png)

Next, configure when the intervention should trigger.

**Python SDK:**

Start by [connecting to Signals](/docs/signals/connection/) to create a `Signals` object. You'll need this connection to publish interventions.

Rule-based interventions use `RuleIntervention` objects. This is the minimum configuration needed to create an intervention definition:

```python
from snowplow_signals import RuleIntervention, InterventionCriterion



hello_intervention = RuleIntervention(

    name="say_hello",

    owner="me@example.com",

    criteria=InterventionCriterion(

        attribute="example_group:test_attribute",

        operator="is not null",

    ),

)



sp_signals.publish([hello_intervention])
```

Once applied and active, this intervention will trigger the first time Signals processes an event that first sets the `example_group` attribute group's `test_attribute` attribute to a value that is not null (e.g. the first time it gets set).

The table below lists all available arguments for a `RuleIntervention`:

| Argument                | Description                                                                                                                                                                   | Type                                                                                                              | Required? |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | --------- |
| `name`                  | The unique name/identifier of the intervention                                                                                                                                | `string`                                                                                                          | ✅         |
| `owner`                 | Email address for the owner of this intervention definition                                                                                                                   | `string`                                                                                                          | ✅         |
| `description`           | A human-readable description of the intervention                                                                                                                              | `string`                                                                                                          | ❌         |
| `version`               | A numeric version for this definition                                                                                                                                         | `integer`                                                                                                         | ❌         |
| `target_attribute_keys` | List of attribute key names to publish this intervention to. If not defined, defaults to the attribute keys associated with any attribute groups you reference in `criteria`. | `string[]`                                                                                                        | ❌         |
| `criteria`              | Tree of `Criterion` expressions to evaluate against attribute key attributes                                                                                                  | One of: `InterventionCriterion`, `InterventionCriteriaAll`, `InterventionCriteriaAny`, `InterventionCriteriaNone` | ✅         |

***

## Criteria

Criteria are the conditional rules that determine when an intervention should trigger.

> **Note:** An intervention is sent only the first time the criteria are met. Read an example of how this works in the [delivery example](/docs/signals/applications/subscribe/#delivery-example).

When a referenced attribute is updated, the updated and previous states are evaluated against the criteria. If the previous state did not meet the conditions but the newly updated state does, the trigger activates. Criteria always refer to the latest published version of the attribute group that contains the attribute.

**Console:**

Defining intervention criteria has three steps:

1. Select which attribute from a published attribute group to evaluate
2. Choose which logical operator to use
3. Enter the value to trigger on

![Intervention criteria configuration showing attribute selection, operator, and value fields](/assets/images/intervention-criteria-attribute-17a46f32fc8d0df79bcee451db3b0b80.png)

When adding more than one criterion, you can require all or any of them to be met.

![Multiple intervention criteria with \&quot;all\&quot; or \&quot;any\&quot; logic selection](/assets/images/intervention-criteria-all-d05e3e972c1f1ec916af4bbee3ac34e3.png)

These attributes are both from groups with the `domain_userid` attribute key. Therefore, this intervention is targeted to users. When a [subscribed](/docs/signals/applications/subscribe/) user reaches 10 page views while using Chrome, the intervention will trigger.

**Python SDK:**

The simplest `criteria` tree takes an `InterventionCriterion` instance, with possible arguments:

| Argument    | Description                                                              | Type                                                                                           |
| ----------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| `attribute` | The attribute group and attribute name to evaluate, separated by a colon | `string`                                                                                       |
| `operator`  | The operator used for evaluating the attribute                           | `str`, see table below                                                                         |
| `value`     | If required by the `operator`, the comparison value                      | `str`, `int`, `float`, `bool`, `list[str]`, `list[int]`, `list[float]`, `list[bool]` or `None` |

This table shows the available operators:

| Operator      | Description                 | Value is required? |
| ------------- | --------------------------- | ------------------ |
| `=`           | Equals                      | ✅                  |
| `!=`          | Not equals                  | ✅                  |
| `<`           | Less than                   | ✅                  |
| `>`           | Greater than                | ✅                  |
| `<=`          | Less than or equal          | ✅                  |
| `>=`          | Greater than or equal       | ✅                  |
| `like`        | Pattern matching (SQL LIKE) | ✅                  |
| `not like`    | Pattern not matching        | ✅                  |
| `rlike`       | Regex pattern matching      | ✅                  |
| `not rlike`   | Regex not matching          | ✅                  |
| `in`          | Value in list               | ✅ (list)           |
| `not in`      | Value not in list           | ✅ (list)           |
| `is null`     | Value is null/empty         | ❌                  |
| `is not null` | Value exists                | ❌                  |

For more complex conditions, `criteria` also accepts `InterventionCriteriaAll`, `InterventionCriteriaAny`, and `InterventionCriteriaNone`. You can use these classes to combine multiple criteria as lists with the following parameters:

| Argument                   | Description                                                                                                  | Parameters                        |
| -------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------- |
| `InterventionCriteriaAll`  | Logically AND all the inner criteria. Only trigger if all the nested criteria are also true/triggered.       | `all`: list of nested `criteria`  |
| `InterventionCriteriaAny`  | Logically OR all the inner criteria. Only trigger if at least one of the nested criteria are true/triggered. | `any`: list of nested `criteria`  |
| `InterventionCriteriaNone` | Logically NOR all the inner criteria. Only trigger if all of the nested criteria are false/untriggered.      | `none`: list of nested `criteria` |

These support nesting, so you can create complex conditions if required:

```python
from snowplow_signals import (

    RuleIntervention,

    InterventionCriterion,

    InterventionCriteriaAll,

    InterventionCriteriaAny,

    InterventionCriteriaNone,

)



criteria = InterventionCriteriaAll(all=[

    InterventionCriterion(attribute="mygroup:pageview_count", operator=">=", value=3),

    InterventionCriteriaAny(any=[...]),

    InterventionCriteriaNone(none=[...]),

])
```

***

## Target and payload

Interventions are delivered to [attribute keys](/docs/signals/concepts/#targeting). By default, the intervention will target the attribute keys of the attribute groups defined in the criteria. Specify attribute keys if you want different targets.

> **Note:** Interventions can be defined against any attribute key, as long as its values are non-enumerable. For example, the built-in Snowplow attribute keys `domain_userid`, `domain_sessionid`, and `network_userid` are suitable targets since their values are canonically formatted UUIDs, e.g. `8c9104e3-c300-4b20-82f2-93b7fa0b8feb`.

Custom targeting can be useful when you have broad criteria but want a more narrow target. For example, if your criteria use a broad attribute like `page_id`, you might not want to send the intervention to everyone on that page. If you're also checking user-specific criteria, target `domain_userid` to reach only the specific user who meets all conditions.

Another use for custom targeting is when you want to target an attribute key that's available in the triggering event, but isn't part of your intervention logic. For example, to send an intervention to a `domain_userid` when they do a certain number of things in a single `pageview_id`. By targeting the user rather than the page, you can ensure that the intervention is received even if they've gone onto a new page by the time the triggering event is processed, or if the application is not subscribed to the `pageview_id`.

**Console:**

Define a custom intervention target scope by selecting attribute keys. These are the attribute keys that will receive the information.

You can also select one or more attribute groups to be sent with the intervention. When the intervention triggers, it will include the latest values for all attributes in the selected groups.

![Intervention delivery configuration showing attribute key targeting and attribute group payload selection](/assets/images/intervention-delivery-8b7a64861da887b3a6be2a28247f5d92.png)

**Python SDK:**

Use the `target_attribute_keys` argument to set custom targets. This example shows how to use criteria and targeting together:

```python
from snowplow_signals import (

    RuleIntervention,

    InterventionCriterion,

    InterventionCriteriaAll,

    LinkAttributeKey,

)



cart_abandonment = RuleIntervention(

    name="cart_abandonment_discount",

    owner="marketing@company.com",

    description="Offer discount when high-value cart is abandoned",

    criteria=InterventionCriteriaAll(all=[

        InterventionCriterion(

            attribute="shopping:cart_value",

            operator=">",

            value=75.00

        ),

        InterventionCriterion(

            attribute="shopping:minutes_since_last_activity",

            operator=">=",

            value=15

        ),

    ]),

    target_attribute_keys=[

        LinkAttributeKey(name="domain_userid")

    ]  # Target individual users

)



sp_signals.publish([cart_abandonment])
```

It will trigger based on attribute changes within the `shopping` attribute group, but only if all conditions are met.

***

## Publish and manage the intervention

Interventions only become active once published to your Signals infrastructure. Signals will then start monitoring attribute value changes. To receive and act on triggered interventions, your application will need an active [subscription](/docs/signals/applications/subscribe/).

**Console:**

Once you're happy with your intervention configuration, click **Create intervention** to save it. It will be saved as a draft, and not yet available to Signals.

![Intervention details page in Snowplow Console showing a status of Draft, with Edit and Publish action buttons](/assets/images/intervention-draft-cec4394a4e720ceae7fb0c2442f089f6.png)

Click the **Edit** button if you want to make changes to the intervention.

To send the intervention configuration to your Signals infrastructure, click the **Publish** button.

![Published intervention page with sample subscription code](/assets/images/intervention-published-937a3576030437ba8a7363b1e8e8e07e.png)

The intervention page also includes sample code to help you subscribe to it. Read more about this on the [subscription](/docs/signals/applications/subscribe/) page.

**Python SDK:**

Use `publish()` to register rule-based interventions with Signals:

```python
sp_signals.publish([hello_intervention])
```

***

### Versioning

Interventions are versioned. This allows you to iterate on the definitions without breaking downstream processes. All interventions start as `v1`.

Within criteria, the attributes are always evaluated based on the latest published version of the attribute group that contains the attribute. For example, if `attribute_group` v1, v2, and v3 all have the required `attribute`, v3 will be used. If you then publish `attribute_group` v4, which removes `attribute`, v3 will still be used.

**Console:**

If you make changes to the definition, the version will be automatically incremented.

**Python SDK:**

Use `version=1` for the first version of an intervention. After publishing, if you want to change the definition in any way, iterate the version number.

***

### Unpublish or delete

To unpublish or delete an intervention in Console, click the `⋮` button on the details page.

![Intervention management menu showing Edit, Unpublish, and Delete options](/assets/images/intervention-edit-delete-ecd541c2c5c565cb3b87489261ced4fb.png)

Unpublishing is version specific. You can republish it later if needed. Choose **Delete** to permanently delete all versions of the intervention.

With the Python SDK, use the `unpublish()` and `delete()` methods.

## Test the intervention

Signals provides two ways to test an intervention in Snowplow Console: a preview you can run before publishing, and a live test you can trigger once the intervention is published.

### Preview

While defining an intervention, you can preview how many users would have received it by running it against the current state of your warehouse data. Click **Run preview** to see how many attribute keys match the criteria you've defined.

![Intervention preview showing test results with match count against scanned warehouse records](/assets/images/intervention-preview-ca15ec102432a285849e6ffcac19d5fa.png)

### Send a test intervention

Once the intervention is published, you can send a test by specifying a particular attribute key identifier. Signals will dispatch an intervention to that identifier so you can verify your [subscription](/docs/signals/applications/subscribe/) is set up correctly and diagnose any connection errors before relying on it in production. Your application must already be subscribed to that attribute key ID to receive the test.

![Intervention connection test showing attribute key input field to send a test intervention](/assets/images/intervention-connection-test-a96363911003583adc095e425a80c5c4.png)

## Direct interventions

Direct interventions have no criteria, and are not tied to attribute values. They use `InterventionInstance` objects, and are pushed to Signals using `push_intervention`, rather than being published like other configuration objects.

> **Note:** Direct interventions are only available using the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) or [Signals API](/docs/signals/connection/#signals-api). They use the API's `api/v1/registry/interventions` endpoints under the hood, which require a valid, authorized bearer token.

You can directly push these interventions to any attribute keys. If the intervention is valid, Signals will immediately deliver it to any [subscribers](/docs/signals/applications/subscribe/) for the targeted attribute key IDs.

```python
from snowplow_signals import AttributeKeyIdentifiers, InterventionInstance, Signals



# The specific attribute key IDs to publish the intervention to

targets = AttributeKeyIdentifiers({

  "domain_sessionid": ["8c9104e3-c300-4b20-82f2-93b7fa0b8feb"],

})



sp_signals.push_intervention(

  targets,

  InterventionInstance(

    name="my_intervention",

    version=1,

  )

)
```

---

# Snowplow Signals Documentation
> Snowplow Signals is a real-time personalization engine that computes and acts on behavioral data from your pipeline. It enables in-session stream and historical user data access for personalized experiences, recommendations, and dynamic pricing.
> Source: https://docs.snowplow.io/docs/signals/introduction/

Snowplow Signals computes user attributes from your behavioral data in real time, and serves them to your applications over an API. Attributes are calculated from your Snowplow event stream as events arrive, or synced from tables in your warehouse.

Use Signals to:

- Personalize content, recommendations, and pricing based on what the user is doing in their current session
- Give chatbots and AI agents context about the user they're interacting with
- Trigger actions automatically when users meet criteria you define, such as offering a discount when a high-value cart is abandoned
- Serve up-to-date behavioral features to ML models, for use cases like lead scoring

If you don't have a Snowplow account yet, sign up for a [Snowplow free trial](https://snowplow.io/get-started/snowplow-free-trial) to try Signals in Snowplow Console.

## Explore Signals

Attributes

Calculate user attributes in real time

[Learn about attributes](/docs/signals/concepts/#attribute-groups) [Define attributes](/docs/signals/attributes/) [Retrieve attributes](/docs/signals/applications/retrieve-attributes/)

Interventions

Trigger actions automatically

[Learn about interventions](/docs/signals/concepts/#interventions) [Define interventions](/docs/signals/interventions/) [Subscribe to interventions](/docs/signals/applications/subscribe/)

Signals Python SDK

Use Signals within a notebook

[Connect to Signals](/docs/signals/connection/#signals-python-sdk) [Define attributes](/docs/signals/attributes/) [Define interventions](/docs/signals/interventions/)

## Get hands on with Signals

> **Tip:** Visit our [Developer Hub](https://snowplow.io/developer-hub/) for demo videos, blog posts, workshops, and more.

[Build a personalized travel agent](/tutorials/signals-personalize-travel/intro/)

[Set up a demo travel site, and integrate with Snowplow Signals to personalize content and chatbot responses based on user behavior](/tutorials/signals-personalize-travel/intro/)

[Score prospects in real time using Signals and ML](/tutorials/signals-ml-prospect-scoring/intro/)

[Try out Signals for live prospect scoring with machine learning](/tutorials/signals-ml-prospect-scoring/intro/)

[Implement real-time interventions in an ecommerce app](/tutorials/signals-interventions/start/)

[Explore Signals interventions in a demo e-shop application](/tutorials/signals-interventions/start/)

[See all tutorials ->](/tutorials/)

---

# Set up Signals
> Enable Signals in Snowplow Console and connect your warehouse to deploy Signals infrastructure alongside your pipeline.
> Source: https://docs.snowplow.io/docs/signals/setup/

Before you can define attributes or interventions, you'll need to enable Signals for your organization. This is a one-time task, done in [Snowplow Console](https://console.snowplowanalytics.com).

If you don't have a Snowplow account yet, sign up for a [Snowplow free trial](https://snowplow.io/get-started/snowplow-free-trial) to experience Signals in Console.

## Enable Signals

Log in to Console and navigate to the **Signals** section. Click **Enable** to start setting up Signals.

![Snowplow Console Signals section showing Enable button to set up connection](/assets/images/console-no-connection-152c34507c46b982e913d47be8f7a082.png)

> **Note:** Signals can also be deployed without connecting to a warehouse. Only Snowflake and BigQuery are supported currently.

You'll need to:

- Select which warehouse to use
- Specify your warehouse account details
- Specify your Snowplow atomic events table
- Run the provided script

Click **Test and create connection** to trigger the Signals deployment. You'll be able to start using Signals as soon as the infrastructure is ready.

## Next steps

Once the infrastructure is ready, navigate to the **Signals** section to manage Signals in the UI. Use the configuration interface to define [attribute groups](/docs/signals/attributes/attribute-groups/), [services](/docs/signals/applications/services/), and [interventions](/docs/signals/interventions/).

![Console Signals landing page with navigation for attribute groups, services, and interventions](/assets/images/console-landing-c09cd5b000aef369b1e855ab7f9294da.png)

To work with Signals programmatically, using the SDKs or API instead of (or alongside) Console, see [Connect to Signals](/docs/signals/connection/).

---

# Adding extra data: the Subject class in C++ tracker
> Attach user and device information to tracked events using the Subject class. Set user ID, screen resolution, viewport, color depth, timezone, language, user-agent, and IP address for richer behavioral data.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/adding-extra-data-the-subject-class/

Subject information describes the user and device associated with the event, such as their user ID, what type of device they used, or what size screen that device had.

Create a subject like this and add it to your tracker using the `Snowplow::create_tracker` call:

```cpp
auto subject = make_shared<Subject>();

subject.set_user_id("a-user-id");



auto tracker = Snowplow::create_tracker("ns", "https://com.acme.collector", POST, "events.db", subject);
```

You can also attach custom Subject information to individual events. In this way, you may track events describing different users or devices using the same tracker. Events can be assigned a shared C++ pointer to a Subject instance using the `set_subject` method. The following example shows how to attach a subject instance to a structured event (see [Tracking specific events](/docs/sources/c-tracker/tracking-specific-events/) for more information on tracking events):

```cpp
auto subject = std::make_shared<Subject>();

subject->set_user_id("another-user");



StructuredEvent se("category", "action");

se.set_subject(subject);
```

The Subject class has a set of `set...()` methods to attach extra data relating to the user to all tracked events. The below table maps the setter functions to the event properties that they affect.

| Setter method           | Resulting event property                   |
| ----------------------- | ------------------------------------------ |
| `set_user_id`           | `user_id`                                  |
| `set_screen_resolution` | `dvce_screenwidth` and `dvce_screenheight` |
| `set_viewport`          | `br_viewwidth` and `br_viewheight`         |
| `set_color_depth`       | `br_colordepth`                            |
| `set_timezone`          | `os_timezone`                              |
| `set_language`          | `br_lang`                                  |
| `set_useragent`         | `useragent`                                |
| `set_ip_address`        | `user_ipaddress`                           |

We will discuss each of these in turn below:

## Set user ID with "set\_user\_id"

You can set the user ID to any string:

```cpp
subject->set_user_id( "{{USER ID}}" );
```

Example:

```cpp
subject->set_user_id("alexd");
```

## Set screen resolution with "set\_screen\_resolution"

If your code has access to the device’s screen resolution, then you can pass this in to Snowplow too:

```cpp
subject->set_screen_resolution( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```cpp
subject->set_screen_resolution(1366, 768);
```

## Set viewport dimensions with "set\_viewport"

If your code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```cpp
subject->set_viewport( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```cpp
subject->set_viewport(300, 200);
```

## Set color depth with "set\_color\_depth"

If your code has access to the bit depth of the device’s color palette for displaying images, then you can pass this in to Snowplow too:

```cpp
subject->set_color_depth( {{BITS PER PIXEL}} );
```

The number should be a positive integer, in bits per pixel. Example:

```cpp
subject->set_color_depth(32);
```

## Set timezone with "set\_timezone"

This method lets you pass a user’s timezone in to Snowplow:

```cpp
subject->set_timezone( {{TIMEZONE}} );
```

The timezone should be a string:

```cpp
subject->set_timezone("Europe/London");
```

## Set the language with "set\_language"

This method lets you pass a user’s language in to Snowplow:

```cpp
subject->set_language( {{LANGUAGE}} );
```

The language should be a string:

```cpp
subject->set_language('en');
```

## Set custom user-agent with "set\_useragent"

To change the [user-agent string](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) sent along with events to identify the application and system, you may set custom useragent using this method:

```cpp
subject->set_useragent( {{USERAGENT}} );
```

The user-agent should be a string:

```cpp
subject->set_useragent("YourApp/5.0 (Macintosh; Intel Mac OS X 10_15_7)");
```

## Set user's IP address with "set\_ip\_address"

To set the user's IP address, you may use this method:

```cpp
subject->set_ip_address( {{IP_ADDRESS}} );
```

The IP address should be a string:

```cpp
subject->set_ip_address("169.254.0.2");
```

If the IP address is not set, the events will be assigned the IP address from the HTTP request by the Collector.

---

# track client sessions with the C++ tracker
> Enable client session tracking in the C++ tracker to track user sessions with foreground and background timeouts. Attach session context entities including session ID, index, and first event timestamp.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/client-sessions/

You can optionally decide to include Client Sessionization. This object will keep track of your users sessions and can be configured to timeout after a certain amount of inactivity.

Activity is determined by how often events are sent with the Tracker – so you will need to send events to keep the current session active. Sessions are updated when new events are tracked. There are two timeouts that are checked: foreground and background timeout. Depending on whether the app is in foreground or backgroud, the relevant timeout is used to compare the time difference since previous event and, in case it surpasses the timeout, a new session is started.

In [Initialisation](/docs/sources/c-tracker/initialisation/), we discussed how to create a tracker with optional session tracking enabled and configurable foreground and background timeouts. If you enabled session tracking when creating the tracker, you can access a `ClientSession` instance using the tracker:

```cpp
auto client_session = Snowplow::get_default_tracker()->get_client_session();
```

To set the background/foreground state you will need to detect this and then set this on the `ClientSession` object like so:

```cpp
client_session.set_is_background(true || false);
```

When client sessions are used, the [`client_session`](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) context entity is added to all tracked event. This entity consists of the following properties:

| Attribute             | Description                                                                                                   | Required? |
| --------------------- | ------------------------------------------------------------------------------------------------------------- | --------- |
| `userId`              | An identifier for the user of the session.                                                                    | Yes       |
| `sessionId`           | An identifier (UUID) for the session.                                                                         | Yes       |
| `sessionIndex`        | The index of the current session for this user.                                                               | Yes       |
| `eventIndex`          | Optional index of the current event in the session. Signifies the order of events in which they were tracked. | No        |
| `previousSessionId`   | The previous session identifier (UUID) for this user.                                                         | No        |
| `storageMechanism`    | The mechanism that the session information has been stored on the device.                                     | Yes       |
| `firstEventId`        | The optional identifier (UUID) of the first event id for this session.                                        | No        |
| `firstEventTimestamp` | Optional date-time timestamp of when the first event in the session was tracked.                              | No        |

## Session store

The session store is used to persist the currently active session.

The tracker provides the `SqliteStorage` class that can be used as the session store. However, you may also provide a custom session store implementation. To do so, define a class that inherits from the `SessionStore` struct:

```cpp
struct SessionStore {

  virtual unique_ptr<json> get_session() = 0;

  virtual void set_session(const json &session_data) = 0;

  virtual void delete_session() = 0;

};
```

The `SessionStore` struct defines functions to retrieve, set, and delete the current session. It represents sessions in their JSON format. The three operations have the following behavior:

| Function         | Description                                                                    |
| ---------------- | ------------------------------------------------------------------------------ |
| `get_session`    | Return a unique pointer to the current session or nullptr if it doesn't exist. |
| `set_session`    | Persist the current session.                                                   |
| `delete_session` | Remove and reset the current session.                                          |

---

# Configure emitters in the C++ tracker
> Configure emitters to send events to your collector with customizable batch sizes, byte limits, and retry behavior. Use SQLite event store or custom EventStore implementation with request callbacks.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/emitters/

Emitters are responsible for sending events to the collector. Each tracker is given a single emitter. Once the emitter receives an event from the Tracker a few things start to happen:

- The event is added to a local SQLite3 database or custom event store (blocking execution).
- A long running daemon thread is started which will continue to send events as long as they can be found in the database (asynchronous).
- The emitter loop will grab a range of events from the database up until the `batch_size` passed to it as configuration.
- The emitter will send all of these events as determined by the Request, Protocol and ByteLimits.
  - Each request is sent in its thread.
- Once sent, it will process the results of all the requests sent and will remove all successfully sent events from the database. If the request failed, the events will be retried after a retry delay (see below).

In [Initialisation](/docs/sources/c-tracker/initialisation/), we discussed how to create a tracker with an emitter configured using `EmitterConfiguration` or by instantiating an `Emitter` instance directly. Both of these options provide the same configuration functionality (e.g., storage options, byte limits, setting custom HTTP clients) that were discussed previously. This page will go into more detail on some of the configurable emitter properties.

## Event store

The event store is used to store an event queue with events scheduled to be sent. Events are added to the event store when they are tracked and removed when they are successfuly emitted or when emitting fails without any scheduled retries.

The tracker provides the `SqliteStorage` class that can be used as the event store. It uses SQLite to store the event queue. By default it will create the required files wherever the application is being run from.

You may also provide a custom event store implementation. To do so, define a class that inherits from the `EventStore` struct:

```cpp
struct EventStore {

  virtual void add_event(const Payload &payload) = 0;

  virtual void get_event_rows_batch(list<EventRow> *event_list, int number_to_get) = 0;

  virtual void delete_event_rows_with_ids(const list<int> &id_list) = 0;

};
```

The `EventStore` struct defines functions to insert, retrieve, and remove events from the queue. Events are represented using their `Payload` instance which is persisted in a `EventRow` wrapper that also assigns IDs to each stored event (these event row IDs are different from event IDs used in the event payloads). There are three supported operations:

| Function                     | Description                                                 |
| ---------------------------- | ----------------------------------------------------------- |
| `add_event`                  | Insert event payload into event queue.                      |
| `get_event_rows_batch`       | Retrieve event rows from event queue up to the given limit. |
| `delete_event_rows_with_ids` | Remove event rows with the given event row IDs.             |

## Emitter request callback

The emitter enables you to set a callback function to be called after events are attempted to be sent to the Collector. This callback is fired after HTTP requests are made and you can subscribe for specific emit statuses. The following statuses can be subscribed to:

- `EmitStatus::SUCCESS` – events were successfuly sent to the Collector.
- `EmitStatus::FAILED_WILL_RETRY` – events failed to be sent to the Collector but will be retried later.
- `EmitStatus::FAILED_WONT_RETRY` – events failed to be sent to the Collector and won't be retried later (they will be dropped).

The callback is given two arguments – list of event IDs, and their emit status. You can only set one callback at once but you can subscribe to multiple emit statuses using binary operations. The following example shows how to set a callback that is called for all three emit statuses and prints to standard output.

```cpp
emitter_configuration.set_request_callback(

    [](list<string> event_ids, EmitStatus emit_status) {

      switch (emit_status) {

      case EmitStatus::SUCCESS:

        printf("Successfuly sent %lu events.\n", event_ids.size());

        break;

      case EmitStatus::FAILED_WILL_RETRY:

        printf("Failed to send %lu events, but will retry.\n", event_ids.size());

        break;

      case EmitStatus::FAILED_WONT_RETRY:

        printf("Failed to send %lu events and won't retry.\n", event_ids.size());

        break;

      }

    },

    EmitStatus::SUCCESS | EmitStatus::FAILED_WILL_RETRY | EmitStatus::FAILED_WONT_RETRY);
```

The callback is executed in a new thread. The `set_request_callback` function can't be called when the Emitter is running.

## HTTP request retry behavior

The Emitter has a retry functionality that repeatedly sends the same events to the Collector in case HTTP requests fail to get through. Requests are retried in case the connection to the Collector fails to be estabilished. They are also retried for all 3xx and 5xx HTTP status codes in server response and most 4xx status codes with the following exceptions – 400, 401, 403, 410, and 422. These status codes signal a rejection of the events being sent to the Collector and, therefore, the events in the requests are dropped and not sent again.

The `Emitter` and `EmitterConfiguration` provide an option to set custom retry behavior for 3xx, 4xx and 5xx HTTP status codes. You can call the `set_retry_for_status_code` function on the `Emitter` or `EmitterConfiguration` instance to define whether to retry or not for a given HTTP status code. Here is an example that overrides the default behavior and enables retries on 422 status code:

```cpp
emitter_configuration.set_retry_for_status_code(422, true);
```

## Request retry delay (back-off)

Failed requests are retried with exponentially increasing delays between subsequent retry requests. This ensures that the Collector is not overwhelmed with too many requests and also saves resources on the client by reducing the number of requests during failures.

The retry delay calculation is an exponential function with multiplicative factor 2. To prevent spikes of traffic, small amount of randomness is added to the delay (at most 10% of the total value). Finally, it is limited to be no larger than around 2 minutes. The following is a sample sequence of the retry delays given 5 failed requests: 0.101s, 0.198s, 0.404s, 0.791s, 1.603s.

## Manual flushing

You may want to force an emitter to send all events in its buffer, even if the buffer is not full. The Tracker class has a `flush()` method which flushes its emitter.

Example:

```cpp
tracker->flush();
```

## Using a custom HTTP Client

You may optionally configure the HTTP client to be used to make HTTP requests to the collector. This is done by passing a unique pointer to a class inheriting from `HttpClient` that the Emitter will take ownership of. If not configured, the Emitter will use the built-in `HttpClientWindows` on Windows, `HttpClientApple` on Apple operating systems, and `HttpClientCurl` on other Unix systems.

---

# C++ tracker SDK
> The Snowplow C++ Tracker enables event tracking from C++ applications, games, and servers. Create subjects, emitters, and trackers to send behavioral events to your Snowplow collector.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/

The [Snowplow C++ Tracker](https://github.com/snowplow/snowplow-cpp-tracker) allows you to track Snowplow events from your C++ apps, games and servers.

There are three basic types of object you will create when using the Snowplow C++ Tracker: subjects, emitters, and trackers.

- A subject represents a user whose events are tracked.
- A tracker constructs events and sends them to an emitter.
- The emitter then sends the event to the endpoint you configure; a valid Snowplow Collector.

---

# Initialize the C++ tracker
> Create a C++ tracker using the Snowplow interface with default or custom configuration. Configure tracker, network, emitter, and session settings using configuration objects or manage Tracker, Emitter, and ClientSession directly.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/initialisation/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

## Import the library

Import the C++ Tracker library and use the `snowplow` namespace like so:

```cpp
#include <snowplow/snowplow.hpp>



using namespace snowplow;
```

That’s it – you are now ready to initialize a tracker instance that you will use to track events. There are 3 ways to do that:

1. Create a tracker with default configuration using the `Snowplow` interface.
2. Create a tracker with custom configuration using the `Snowplow` interface.
3. Create and manage `Tracker`, `Emitter`, and `ClientSession` directly.

## Option 1: Creating a tracker with default configuration using the "Snowplow" interface

The `Snowplow` class provides static methods that let you easily create a new tracker. It can be as simple as:

```cpp
auto tracker = Snowplow::create_tracker(

    "ns", // tracker namespace used to identify the tracker

    "https://com.acme.collector", // Snowplow collector URL

    POST, // HTTP method used to send events to the collector

    "events.db" // Relative path to an SQLite database used for event queue and session tracking

);
```

Optionally, you may choose to attach a `Subject` instance with information about the user and device (see the next page on [Adding data](/docs/sources/c-tracker/adding-extra-data-the-subject-class/) to learn more about the `Subject`):

```cpp
auto subject = std::make_shared<Subject>() // initialize a C++ shared pointer for the Subject

subject->set_user_id("a-user-id");



auto tracker = Snowplow::create_tracker("ns", "https://com.acme.collector", POST, "events.db", subject);
```

Finally, client session tracking will be automatically enabled and session information will be attached to all events (see documentation on [Client sessions](/docs/sources/c-tracker/client-sessions/) for more details). If you wish to disable session tracking, pass `false` as the last argument:

```cpp
auto tracker = Snowplow::create_tracker("ns", "https://com.acme.collector", POST, "events.db", subject, false);
```

After you create a tracker, you can access it from anywhere using it's namespace:

```cpp
auto tracker = Snowplow::get_tracker("ns");
```

You can also access the default tracker using `Snowplow::get_default_tracker()`. The default tracker is the first initialized tracker (unless it is removed). If you have multiple trackers, you can choose the default tracker by calling `Snowplow::set_default_tracker(tracker2)`.

To remove reference of previously initialized tracker from the Snowplow interface (the tracker will be deleted once all remaininig references to it are deleted):

```cpp
Snowplow::remove_tracker(tracker);
```

## Option 2: Creating a tracker with custom configuration using the "Snowplow" interface

There are a number of settings that you may want to customize when creating a tracker instance. You can make use of configuration objects to provide custom configuration for the tracker:

```cpp
TrackerConfiguration tracker_config(

    "namespace", // tracker namespace

    "app-id", // application ID

    mob // platform that the tracker runs on

);

tracker_config.set_desktop_context(false); // do not attach a desktop context entity to events (on by default)



NetworkConfiguration network_config("https://com.acme.collector", POST); // collector URL and method



EmitterConfiguration emitter_config("sp.db"); // event queue DB path if SQLite or custom `EventStore` instance

emitter_config.set_batch_size(500); // maximum number of events to send at a time



SessionConfiguration session_config(

    "sp.db", // session DB path if SQLite (usually same as event queue) or custom `SessionStore` instance

    5000, // foreground timeout in ms

    5000 // background timeout in ms

);



auto tracker = Snowplow::create_tracker(tracker_config, network_config, emitter_config, session_config);
```

### Tracker configuration using "TrackerConfiguration"

`TrackerConfiguration` contains settings to identify and configure the tracker. It's constructor takes 3 attributes:

| Constructor attribute | Description                                                                                          | Default |
| --------------------- | ---------------------------------------------------------------------------------------------------- | ------- |
| `name_space`          | Tracker namespace to identify the tracker and also attach as a property to tracked events.           | None    |
| `app_id`              | Application ID.                                                                                      | ""      |
| `platform`            | Enum of the platform the Tracker is running on, can be one of: web, mob, pc, app, srv, tv, cnsl, iot | srv     |

It further provides 2 optional setter functions:

| Setter                | Description                                                                                                             | Default |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------- |
| `set_use_base64`      | Whether to use base64 encoding in events.                                                                               | true    |
| `set_desktop_context` | Whether to add a desktop\_context, which gathers information about the device the tracker is running on, to each event. | true    |

### Network configuration using "NetworkConfiguration"

`NetworkConfiguration` has only two properties set in it's constructor to configure the Snowplow collector:

| Constructor attribute | Description                                                                                                                                                                   | Default                                                                                   |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `collector_url`       | Full URL of the Snowplow collector including the protocol (or defaults to HTTPS if protocol not present).                                                                     | None                                                                                      |
| `method`              | HTTP method to use when sending events to collector – GET or POST.                                                                                                            | POST                                                                                      |
| `curl_cookie_file`    | Path to a file where to store cookies in case http\_client is nullptr and the CURL HTTP client is used – only relevant under Linux (CURL is not used under Windows and macOS) | In-memory cookie storage with CURL on Linux, platform native storage on Windows and macOS |

Additionally, it provides the following setter functions:

| Setter            | Description                                                                | Default                           |
| ----------------- | -------------------------------------------------------------------------- | --------------------------------- |
| `set_http_client` | Unique pointer to a custom HTTP client to send GET and POST requests with. | Platform-specific implementation. |

### Emitter configuration using "EmitterConfiguration"

`EmitterConfiguration` brings additional settings for the constructor. It provides two constructors that accept the event store either as a path to the SQLite database or a custom `EventStore` object. The following configurations are identical:

```cpp
EmitterConfiguration config1("sp.db");



auto storage = std::make_shared<SqliteStorage>("sp.db");

EmitterConfiguration config2(storage); // you can also pass a custom `EventStore`
```

Additionally, it provides the following setter functions:

| Setter                             | Description                                                                                                                                  | Default     |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| `set_batch_size`                   | The maximum amount of events to send at a time.                                                                                              | 250 events  |
| `set_byte_limit_get`               | The byte limit when sending a GET request.                                                                                                   | 40000 bytes |
| `set_byte_limit_post`              | The byte limit when sending a POST request.                                                                                                  | 40000 bytes |
| `set_request_callback`             | Set a callback to call after emit requests are made with the resulting emit status (see page about Emitter for more info).                   | None        |
| `set_custom_retry_for_status_code` | Set a custom retry rule for when the HTTP status code is received in emit response from Collector (see page about Emitter for more details). | None        |

### Session configuration using "SessionConfiguration"

`SessionConfiguration` is an optional attribute in `create_tracker`. If passed, session tracking will be enabled with the given configuration. If not passed, session tracking will not be enabled. The constructor takes 3 attributes:

| Constructor attribute        | Description                                                                                                                                                                         | Default    |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `session_store` or `db_name` | You may either pass a path to an SQLite database to be used for session storage or a custom implementation of `SessionStore` similar as for `EventStore` in `EmitterConfiguration`. | None       |
| `foreground_timeout`         | Timeout in ms for updating the session when the app is in background.                                                                                                               | 30 minutes |
| `background_timeout`         | Timeout in ms for updating the session when the app is in foreground.                                                                                                               | 30 minutes |

## Option 3: Managing "Tracker", "Emitter", and "ClientSession" directly

The third option to initialise a new tracker is to instantiate it and the related components directly. This option is suitable in case you want supply a custom emitter or client session implementation. If you don't want to do that, we recommend using Option 2 which gives you the same configuration options with a simpler API.

The following example takes you through the steps needed to initialize a storage, emitter, subject, client session, and tracker:

```cpp
#include <snowplow/snowplow.hpp>



// 1. create storage for event queue and session information

auto storage = std::make_shared<SqliteStorage>("sp.db");

// 2. create an emitter for sending event to Snowplow collector

auto emitter = std::make_shared<Emitter>(storage, "com.acme.collector");

// 3. create a subject with user and device information

auto subject = std::make_shared<Subject>();

subject->set_user_id("a-user-id");

// 4. optionally create a client_session for session tracking

auto client_session = std::make_shared<ClientSession>(storage, 5000, 5000);

// 5. finally, create the tracker instance

auto tracker = std::make_shared<Tracker>(emitter, subject, client_session, "pc", "app_id", "ns");
```

Once you initialize the tracker, you can optionally register it with the Snowplow interface:

```cpp
Snowplow::register_tracker(tracker);
```

This will make accessible from anywhere using `Snowplow::get_tracker("ns")`.

The following will explain the individual components shown in the above code sample in more detail.

### Storage

The storage has two functions in the example above – it is used by the `emitter` to persist an event queue with events to be sent, and it is used by the `client_session` to persist the current session. The tracker provides an SQL storage (`SqliteStorage`) implementation, but you may introduce your own storage as described in ["Emitters"](/docs/sources/c-tracker/emitters/) and ["Client Sessions"](/docs/sources/c-tracker/client-sessions/).

### Emitter

Emitter is a required component responsible for sending tracked events to the collector.

Accepts an argument of an Emitter instance pointer; if the object is `NULL` will throw an exception. See Emitters for more on emitter configuration.

| **Argument Name**  | **Description**                                                                                                                                                               | **Required?** | **Default**                                                                               |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------- |
| `event_store`      | Defines the database to use for event queue                                                                                                                                   | Yes           | –                                                                                         |
| `uri`              | The collector URI (excluding protocol) to send events to                                                                                                                      | Yes           | –                                                                                         |
| `method`           | The request type to use (GET or POST)                                                                                                                                         | No            | POST                                                                                      |
| `protocol`         | The protocol to use (HTTP or HTTPS)                                                                                                                                           | No            | HTTPS                                                                                     |
| `batch_size`       | The maximum amount of events to send at a time                                                                                                                                | No            | 250 events                                                                                |
| `byte_limit_post`  | The byte limit when sending a POST request                                                                                                                                    | No            | 40000 bytes                                                                               |
| `byte_limit_get`   | The byte limit when sending a GET request                                                                                                                                     | No            | 40000 bytes                                                                               |
| `http_client`      | Unique pointer to a custom HTTP client to send GET and POST requests with                                                                                                     | No            | Platform-specific implementation.                                                         |
| `curl_cookie_file` | Path to a file where to store cookies in case http\_client is nullptr and the CURL HTTP client is used – only relevant under Linux (CURL is not used under Windows and macOS) | No            | In-memory cookie storage with CURL on Linux, platform native storage on Windows and macOS |

### Subject

The user which the Tracker will track. Accepts an argument of a Subject instance pointer.

You don’t need to set this during Tracker construction; you can use the `tracker.set_subject(...)` method afterwards. In fact, you don’t need to create a subject at all. If you don’t, though, your events won’t contain user-specific data such as timezone and language.

### Client session

The client sessions which the Tracker will attach to each event. Accepts an argument of a ClientSession instance pointer.

Adds the ability to attach a ClientSession context to each event that leaves the Tracker. This object will persistently store information about the sessions that have occurred for the life of the application – unless the database is destroyed.

### Tracker

The tracker instance lets you track events. You should maintain a reference to the initialized tracker during the app lifetime.

| **Argument Name** | **Description**                                                                                                                           | **Required?** | **Default** |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------- |
| `emitter`         | The emitter to which events are sent                                                                                                      | Yes           | –           |
| `subject`         | The user being tracked                                                                                                                    | No            | nullptr     |
| `client_session`  | Client session recording                                                                                                                  | No            | nullptr     |
| `platform`        | The platform the Tracker is running on                                                                                                    | No            | "srv"       |
| `app_id`          | The application ID                                                                                                                        | No            | ""          |
| `name_space`      | The namespace of the tracker instance used to identify the tracker.                                                                       | No            | ""          |
| `use_base64`      | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64). Defaults to true to ensure that no data is lost or corrupted. | No            | true        |
| `desktop_context` | Whether to add a `desktop_context` to events                                                                                              | No            | true        |

The `desktop_context` gathers extra information about the device it is running on and sends it along with every event that is made by the Tracker.

An example of the data in this context:

```json
{

    "deviceManufacturer": "Apple Inc.",

    "deviceModel": "MacPro3,1",

    "deviceProcessorCount": 8,

    "osIs64Bit": true,

    "osServicePack": "",

    "osType": "macOS",

    "osVersion": "10.11.2"

}
```

For more information the raw JsonSchema can be found [here](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.snowplow/desktop_context/jsonschema/1-0-0).

---

# Install the C++ tracker
> Install the C++ tracker using CMake, FetchContent, or by copying source files. Supports macOS, Windows, and Linux with C++ 11 minimum. Configure dependencies including SQLite3, CURL, and LibUUID.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/setup/

The Snowplow C++ Tracker has been built and tested using C++ 11 as a minimum.

Supported operating systems:

- macOS
- Windows
- Linux

## Installing in your project

There are three ways to install the tracker in your app:

1. By adding the project into your `CMakeLists.txt` as a subdirectory.
2. By installing the project and importing it into your app using CMake's `find_package` command.
3. By copying source files inside the `include` folder into your codebase.

### As an imported target in your CMake project

First, build and install the project. Make sure the project uses the external JSON libraries (`SNOWPLOW_USE_EXTERNAL_JSON=ON`). If you're building a static library (`SNOWPLOW_USE_EXTERNAL_SQLITE=ON`) you also need to use SQLite3 as an external library (`SNOWPLOW_USE_EXTERNAL_SQLITE=ON`).

If you have `SQLite3`, `CURL` or `LibUUID` available as system libraries but you need to use them from a different package (e.g. from Conan) you need to set `CMAKE_FIND_PACKAGE_PREFER_CONFIG=ON` to prevent linking to the system libraries.

```cmake
cmake [...] -DCMAKE_INSTALL_PREFIX=[...]

    -DSNOWPLOW_USE_EXTERNAL_JSON=ON -DSNOWPLOW_USE_EXTERNAL_SQLITE=ON \

    -DCMAKE_FIND_PACKAGE_PREFER_CONFIG=ON \

    -DSNOWPLOW_BUILD_TESTS=0 -DSNOWPLOW_BUILD_EXAMPLE=0 -DSNOWPLOW_BUILD_PERFORMANCE=0
```

After building and installing the project you can use `find_package` to import it into your `CMakeLists.txt`:

```cmake
find_package(snowplow REQUIRED CONFIG)

...

target_link_libraries(your-target snowplow::snowplow)
```

Make sure your project finds the same dependencies what was visible for Snowplow when you were building and installing it. For example, if you have both system and local SQlite3 installations and `CMAKE_FIND_PACKAGE_PREFER_CONFIG` was `ON` for Snowplow but `OFF` for your project, Snowplow will be built with the local SQLite3 while during `find_package(snowplow)` in your project it will find the system one.

### As a subdirectory in your CMake project

Cmake version 3.15 or greater is required. You may add the library to your project target (`your-target`) using `FetchContent` like so:

```cmake
include(FetchContent)

FetchContent_Declare(

  snowplow

  GIT_REPOSITORY https://github.com/snowplow/snowplow-cpp-tracker

  GIT_TAG        2.0.0

)

FetchContent_MakeAvailable(snowplow)

target_link_libraries(your-target snowplow)
```

### Copying files to your project

Download the most recent release from the [releases section](https://github.com/snowplow/snowplow-cpp-tracker/releases). Everything in the `include` folder will need to be included in your application.

The project has two dependencies that need to be included in your project: [nlohmann/json](https://github.com/nlohmann/json) and [the amalgamated version of sqlite3](https://www.sqlite.org/download.html). You will need to update the include paths in headers `include/snowplow/thirdparty/json.hpp` and `include/snowplow/thirdparty/sqlite3.hpp`.

### Additional requirements under Linux

Additionally, under Linux, the following libraries need to be installed:

- curl (using `apt install libcurl4-openssl-dev` on Ubuntu)
- uuid (using `apt install uuid-dev` on Ubuntu)

---

# Track specific events with the C++ tracker
> Track custom behavioral events using SelfDescribingEvent, StructuredEvent, ScreenViewEvent, and TimingEvent. Add custom context entities and true timestamps to events sent to your Snowplow collector.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/tracking-specific-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in event classes to help you track different kinds of events. When instantiated, their objects can be passed to the `tracker.track()` methods to send events to the Snowplow collector. The event classes range from single purpose ones, such as `ScreenViewEvent`, to the more complex but flexible `SelfDescribingEvent`, which can be used to track any kind of user behavior. We strongly recommend using `SelfDescribingEvent` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

The tracker provides the following event classes for tracking events out of the box:

| **Function**          | **Description**                                         |
| --------------------- | ------------------------------------------------------- |
| `StructuredEvent`     | Tracks a Snowplow custom structured event               |
| `SelfDescribingEvent` | Tracks a Snowplow custom unstructured event             |
| `ScreenViewEvent`     | Tracks the user viewing a screen within the application |
| `TimingEvent`         | Tracks a timing event                                   |

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

## Custom event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event's entities together form the event context. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a screen view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Each tracking type provides a `set_context()` method which accepts a vector of context entities:

```cpp
vector<SelfDescribingJson> context;
```

If a visitor arrives on a page advertising a movie, the dictionary for a single custom context entity in the list might look like this:

```cpp
vector<SelfDescribingJson> context;

SelfDescribingJson sdj(

  "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

  "{\"movie_name\":\"Solaris\",\"poster_country\":\"JP\"}"_json

);

context.push_back(sdj);
```

This is how to fire a structured event with the above custom context entity:

```cpp
StructuredEvent se("category", "action");

se.set_context(context);



Snowplow::get_default_tracker()->track(se);
```

Note that even though there is only one custom context entity attached to the event, it still needs to be placed in a list.

**INFO**: We use the excellent json library from Github community member [Niels (nlohmann)](https://github.com/nlohmann) for all JSON parsing. For more information on using this library please visit the [Technical information on Github](https://github.com/nlohmann/json).

## Optional true-timestamp argument

Each event class supports an optional true-timestamp argument; this allows you to provide the true-timestamp attached to this event to help with the timing of events in multiple timezones. The timestamp should be in milliseconds since the Unix epoch.

Here is an example tracking a structured event and supplying the optional true-timestamp argument:

```cpp
StructuredEvent se("category", "action");



// As it is optional you will need to pass the address for this value

unsigned long long true_tstamp = "1368725287000";

se.set_true_timestamp(&true_tstamp);



Snowplow::get_default_tracker()->track(se);
```

## Track SelfDescribing/Unstructured events with "SelfDescribingEvent"

Use the `SelfDescribingEvent` type to track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

`SelfDescribingEvent` provides the following properties:

| **Argument** | **Description**             | **Required?** | **Validation**     |
| ------------ | --------------------------- | ------------- | ------------------ |
| `event`      | The properties of the event | Yes           | SelfDescribingJson |

Example:

```cpp
// Create a JSON containing your data

json data = "{\"level\":5,\"saveId\":\"ju302\",\"hardMode\":true}"_json;



// Create a new SelfDescribingJson

SelfDescribingJson sdj("iglu:com.example_company/save-game/jsonschema/1-0-2", data);



SelfDescribingEvent sde(sdj);

Snowplow::get_default_tracker()->track(sde);
```

For more on JSON schema, refer to [this page](/docs/fundamentals/schemas/).

## Track screen views with "ScreenViewEvent"

Use the `ScreenViewEvent` type to track a user viewing a screen (or equivalent) within your app. This is the page view equivalent for apps that are not webpages.

| **Argument** | **Description**                     | **Required?** | **Type** |
| ------------ | ----------------------------------- | ------------- | -------- |
| `name`       | Human-readable name for this screen | No            | \*string |
| `id`         | Unique identifier for this screen   | No            | \*string |

Although name and id are not individually required, at least one must be provided or the event will fail validation and subsequently throw an exception.

The event uses the following Iglu schema: [iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0).

Example:

```cpp
string name = "Screen ID - 5asd56";



ScreenViewEvent sve;

sve.name = &name;



Snowplow::get_default_tracker()->track(sve);
```

## Track structured events with `StructuredEvent`

Use the `StructuredEvent` type to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required). This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| **Argument** | **Description**                                                  | **Required?** | **Validation** |
| ------------ | ---------------------------------------------------------------- | ------------- | -------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes           | string         |
| `action`     | Defines the type of user interaction which this event involves   | Yes           | string         |
| `label`      | A string to provide additional dimensions to the event data      | No            | \*string       |
| `property`   | A string describing the object or the action performed on it     | No            | \*string       |
| `value`      | A value to provide numerical data about the event                | No            | \*double       |

Example:

```cpp
StructuredEvent se("shop", "add-to-basket"); // constructor takes category and action

se.property = "pcs";

se.value = 25.6;



Snowplow::get_default_tracker()->track(se);
```

## Track timing events with "TimingEvent"

Use the `TimingEvent` type to track user timing events such as how long resources take to load.

Its properties are as follows:

| **Argument** | **Description**           | **Required?** | **Validation** |
| ------------ | ------------------------- | ------------- | -------------- |
| `category`   | The category of the event | Yes           | \*string       |
| `variable`   | The variable of the event | Yes           | \*string       |
| `timing`     | The timing of the event   | Yes           | \*int64        |
| `label`      | The label of the event    | No            | \*string       |

The event uses the following Iglu schema: [iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0).

Example:

```cpp
TimingEvent te("category", "variable", 123);

Snowplow::get_default_tracker()->track(te);
```

---

# Upgrade to newer versions of the C++ tracker
> Migration instructions for upgrading to newer C++ tracker versions. Learn about breaking changes in v2.0.0, v1.0.0, v0.3.0, and v0.2.0 including CMake updates and API changes.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/upgrading-to-newer-versions/

This page gives instructions for upgrading to newer versions of the C++ tracker.

## Upgrading to v2.0.0

There have been the following breaking changes:

- CMake minimum version changed 3.14 -> 3.15
- libcurl has been removed as a dependency of Snowplow on macOS and the CURL HTTP client is no longer available on macOS. If a project that is using Snowplow as a subdirectory relies on getting libcurl through Snowplow then it will fail now.
- Also, sqlite3, libcurl, libuuid are now private dependencies. If a project relies on getting the sqlite3/libcurl/libuuid include paths through the Snowplow tracker library, then it will fail now.

## Upgrading to v1.0.0

The package added support for the cmake build system. You may make use of this and import the package in your cmake build configuration. Importing source code by copying files is still supported. However, please note that the code was moved from the `src` folder to the `include` folder and you will need to import the sqlite3 and nlohmann/json dependencies on your own and update their location in `include/snowplow/thirdparty/sqlite3.hpp` and `include/snowplow/thirdparty/json.hpp` files. See instructions on [the Setup page](/docs/sources/c-tracker/setup/) for more information.

This version changed how the `Tracker` and `Emitter` instances are initialized and introduced the `Snowplow::create_tracker()` API. Please refer to [the page about](/docs/sources/c-tracker/initialisation/) initialization to learn more about the new API. The `Tracker::init()` calls should be replaced with `Snowplow::create_tracker()` and `Tracker::instance()` with `Snowplow::get_tracker(namespace)` or `Snowplow::get_default_tracker()`.

Also note that the default protocol has changed from HTTP to HTTPS.

## Upgrading to v0.3.0

There are a few breaking changes in this release. Please make sure to:

1. Remove the `Tracker::` prefix when referring to event types, e.g., use `ScreenViewEvent()` instead of `Tracker::ScreenViewEvent()`.
2. Use the common `tracker->track(event)` function to track events (instead of `tracker->track_self_describing_event(event)`, `tracker->track_screen_view(event)`, ...).
3. Use the `event.set_context(context)` and `event.set_true_timestamp(tt)` setters instead of accessing `event.contexts` and `event.true_timestamp` directly.
4. Event IDs are no longer accessible from the event objects but are returned from the `tracker->track(event)` function.
5. `Emitter` and `ClientSession` no longer accept database path string for storage but require an instance of `SqliteStorage` (or other storage implementation, see the docs).
6. Use `#include "snowplow.hpp"` to import all public APIs of the tracker instead of including individual files.

## Upgrading to v0.2.0

This version adds a `snowplow` namespace to all tracker components. You will need to import specific or all types from the namespace:

```cpp
using namespace snowplow;
```

The constructor for `ClientSession` also changed, dropping the last argument for `check_interval` that is no longer necessary. Simply remove the last argument when creating an instance of `ClientSession`.

---

# CDN trackers
> Track page views at the CDN level to capture visits from bots, AI agents, and other clients that don't run JavaScript.
> Source: https://docs.snowplow.io/docs/sources/cdn-trackers/

You might want to track page views by bots and AI agents to understand how they consume your content. Regular [web tracking](/docs/sources/web-trackers/), however, does not capture these events because many bots and agents don't execute JavaScript.

If your website is deployed through a CDN, you can track events at that level, which would include _all_ requests.

```mermaid
flowchart LR
  agent(Agent) --> cdn(CDN) -->|serve| website(Website content)
  cdn -.->|track request| snowplow(Snowplow)
```

> **Note:** CDN-level tracking is complementary to client-side web tracking. While it surfaces traffic invisible to client-side tracking, it has its own blind spots.
>
> For example, for single-page applications (SPAs), client-side tracking will correctly capture multiple page views. At the CDN level, the same set of page views would correspond to only a single request.

## Cloudflare

You can add Snowplow tracking to Cloudflare via a [Cloudflare Worker](https://developers.cloudflare.com/workers/). Depending on your monthly number of requests, you will likely require a [Workers Paid plan](https://developers.cloudflare.com/workers/platform/pricing/).

Follow Cloudflare documentation to:

- [Create a worker](https://developers.cloudflare.com/workers/get-started/dashboard/) and add the following script
- [Route your website to the worker](https://developers.cloudflare.com/workers/configuration/routing/routes/#set-up-a-route-in-the-dashboard)

If you are already using Workers, you will need to add the code to your existing worker script.

> **Note:** See the comments in the script for various parts you need to customize.

```javascript
// Add your Snowplow collector domain name here.

// IMPORTANT: Do not use domains registered via Cloudflare,

//            as that can lead to DNS resolution issues inside a worker script.

//            CDI customers can opt for the <name>.collector.snplow.net domain instead.

const collectorUrl =

  "https://<collector-domain>/com.snowplowanalytics.snowplow/tp2";



// Headers to forward to Snowplow.

const forwardHeaders = ["referer", "signature-agent", "signature-input", "signature"];



// App id to include with the events.

const appId = "my-app"



// A helper function to decide which requests to track.

// If you track everything, you will receive Snowplow events for requests to static assets, including images, fonts, etc. This is probably too much.

// The example below filters it to just page requests and /llms.txt.

function shouldTrackRequest(pathname) {

  const dotIndex = pathname.lastIndexOf('.');

  if (dotIndex === -1 || dotIndex < pathname.lastIndexOf('/')) return true;

  return pathname.endsWith('llms.txt');

}



async function trackRequest(request) {

  const headers = {

    "content-type": "application/json",

    // This prevents the tracking of the user or agent IP address.

    // We recommend anonymous tracking for two reasons:

    //   1) There is no mechanism for the user to opt out,

    //      since the IP address will be collected upon the very first visit.

    //   2) With a focus on bot/agent traffic, IP address is not very relevant.

    "sp-anonymous": "*",

  };

  for (const name of forwardHeaders) {

    const value = request.headers.get(name);

    if (value) headers[name] = value;

  }



  const payload = {

    schema: "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",

    data: [

      {

        e: "pv",

        ua: request.headers.get("user-agent"),

        aid: appId,

        p: "srv",

        tv: "cf-worker-1.0.0",

        url: request.url,

      },

    ],

  };



  await fetch(collectorUrl, {

    method: "POST",

    headers,

    body: JSON.stringify(payload),

  });

}



export default {

  async fetch(request, env, ctx) {

    const url = new URL(request.url);

    const response = await fetch(request);



    if (shouldTrackRequest(url.pathname)) {

      ctx.waitUntil(trackRequest(request));

    }



    return response;

  },

};
```

> **Note:** The [`ctx.waitUntil`](https://developers.cloudflare.com/workers/runtime-apis/context/#waituntil) call ensures that the request is tracked. It does not add extra latency or block the response to the client.

You will receive the following fields in the events (as well as all relevant fields added by enrichments, e.g., `page_urlpath`, `page_urlquery`):

| Field       | Value                     | Description                                                         |
| ----------- | ------------------------- | ------------------------------------------------------------------- |
| `event`     | `page_view`               | Event type: page view                                               |
| `platform`  | `srv`                     | Platform: server-side, to distinguish from browser events           |
| `app_id`    | your app id               | Application identifier, so you can filter these events in your data |
| `v_tracker` | `cf-worker-1.0.0`         | Tracker version                                                     |
| `useragent` | from the original request | The visitor's `User-Agent` header                                   |
| `page_url`  | from the original request | The URL of the requested page                                       |

---

# First-party tracking
> Configure a custom collector domain matching your primary domain to enable first-party cookies and bypass browser tracking limitations.
> Source: https://docs.snowplow.io/docs/sources/first-party-tracking/

This feature lets you configure a custom collector domain to match your primary domain, enabling first-party cookies.

When your collector domain (e.g. `collector.snwplow.net`) does not match your primary domain (e.g. `flowershop.ai`), web browsers can impose limitations on the collected data. For example, cookies set by the collector might be only kept for 7 days.

With first-party tracking, you can configure a custom collector domain (e.g. `c.flowershop.ai`) to match your primary domain (e.g. `flowershop.ai`), sidestepping these limitations.

Note that in light of the [latest ITP restrictions](https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense), you will also need to [use a Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) to fully persist the cookies.

> **Info:** Before starting, ensure you can access and edit the configuration of your hosting/DNS provider. This is necessary to complete the implementation.

> **Info:** The flow described below might differ depending on the version of Snowplow you are using.

## Selecting the domain names

When configuring first-party tracking, you first need to set a collector domain where the events will be sent to by the web browser. This must be a subdomain of your primary domain. For example, for `flowershop.ai` it could be `t.flowershop.ai` or `c.app.flowershop.ai`.

> **Tip:** Avoid words such as “track”, “collect” or “analytics”, as ad blockers may prevent data from being sent to domains containing these words.

You also need to select a cookie domain. This will determine on which websites the cookie will be available. For example, if your cookie domain is `flowershop.ai`, the cookie will be available on `flowershop.ai`, as well as any subdomains.

> **Note:** For Snowplow collector to be able to set the cookie, the cookie domain must be a parent domain of the collector domain.

Here are a few examples:

| Collector domain      | Cookie domain       | Cookie available on                                |
| --------------------- | ------------------- | -------------------------------------------------- |
| `t.flowershop.ai`     | `flowershop.ai`     | `flowershop.ai`, `app.flowershop.ai`, etc          |
| `c.app.flowershop.ai` | `app.flowershop.ai` | `app.flowershop.ai`, `beta.app.flowershop.ai`, etc |

When you enter a collector domain, we will automatically suggest a cookie domain. You can edit it to suit your use case.

**How to choose the cookie domain**

Setting the cookie domain to your primary domain has the benefit of allowing you to track users across any subdomains. For example, a cookie domain of `flowershop.ai` would work across all the following subdomains (and would allow you to track the same user ids):

- `docs.flowershop.ai`
- `app.flowershop.ai`
- `test.app.flowershop.ai`
- `flowershop.ai`

In some cases, you may wish to separate the tracking of user behavior for different subdomains.

For example, if you own both `gardening.primary-domain.co.uk` and `insurance.primary-domain.co.uk` and only want to track users on `gardening.primary-domain.co.uk`, you can select that as your cookie domain and `c.gardening.primary-domain.co.uk` as your collector domain.

![The Custom domain form showing a Domain field set to \&quot;t.flowershop.ai\&quot; and a Cookie domain field auto-populated with \&quot;flowershop.ai\&quot;](/assets/images/Screenshot_enter_domain-f90bda00daea98a474af1e6bb6a0881d.png)

## Configuring DNS records

In the next step, Snowplow will generate the required DNS records. This may take several minutes.

When the records are ready, you will receive a confirmation by email.

![The Create DNS records screen showing a loading state with the message \&quot;We\&#39;re generating the configuration settings that you need to apply into your DNS provider\&quot;](/assets/images/Screenshot_create_dns_records-5c11e37bafecca032938c1d40f8025d4.png)

Once the DNS records are available, copy them into your domain provider.

The set of DNS records will contain a special record that allows Snowplow to verify that the setup is correct.

> **Info:** You will have to conclude this step within 72 hours. If Snowplow is unable to verify the setup within that timeframe, you will have to restart the process from the beginning.

Once Snowplow verifies your DNS setup, the status in the bottom left will change and you will receive a confirmation by email.

![The Custom domain details page showing domain \&quot;t.cromah.com\&quot;, cookie domain \&quot;cromah.com\&quot;, a DNS records table with one CNAME and two A records, and a green status indicator reading \&quot;First-party server tracking on cromah.com and its sub-domains\&quot;](/assets/images/Screenshot_dns_records-fd5ce76357c4e4726d9032ce85f974d3.png)

## Updating your tracking code

Once the new custom domain is verified, you can implement the tracking code.

This can be done in several ways:

- If you have not yet implemented tracking and wish to implement the JavaScript tracking snippet, navigate to “Start tracking events”. You can select the new custom domain from the dropdown and the code snippet will automatically update.
- If you already have tracking implemented and are using the JavaScript tracker, follow the above step or update the collector URL manually.
- If you wish to implement tracking or already have tracking with one of our other trackers then you will need to manually change the collector URL.

![The \&quot;Select a custom domain\&quot; dropdown with \&quot;t.cromah.com\&quot; selected, and the JavaScript tracking snippet below showing the collector URL set to the custom domain](/assets/images/Screenshot_implement_new_tracking_snippet-fcb7544f202315342935e5967e2ba8a9.png)

---

# Global context for Flutter tracker
> Add static global context to all Flutter tracker events. Configure global context at initialization or add and remove it at runtime using tags.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/adding-data/global-context/

**Global context** lets you define context entities once and have them sent with every tracked event. This saves you from having to manually attach the same context entities to every individual event call.

Here is an example that adds a global context entity to all subsequently tracked events:

```dart
// Create a context entity

const userContext = SelfDescribing(

  schema: 'iglu:com.my_company/user/jsonschema/1-0-0',

  data: {

    'userId': 'abc123',

    'plan': 'premium'

  }

);



// Add it as a global context entity — it will be attached to all events

await tracker.addGlobalContexts('user_context_tag', userContext);



// This event will include the global entity automatically

tracker.track(ScreenView(name: 'home_screen'));
```

## Configure global context at initialization

You can set up static global context when you initialize the tracker. This is the most straightforward approach if you have context entities that should be present for the entire lifetime of the tracker.

Use `GlobalContextsConfiguration` and pass it to `Snowplow.createTracker()`:

```dart
SnowplowTracker tracker = await Snowplow.createTracker(

  namespace: 'ns1',

  endpoint: 'http://your-collector-endpoint.com',

  globalContextsConfig: const GlobalContextsConfiguration(

    contexts: [

      SelfDescribing(

        schema: 'iglu:com.my_company/user/jsonschema/1-0-0',

        data: {

          'userId': 'user-12345',

          'plan': 'premium'

        }

      ),

      SelfDescribing(

        schema: 'iglu:com.my_company/app_environment/jsonschema/1-0-0',

        data: {

          'environment': 'production',

          'release_version': '1.0.0'

        }

      )

    ]

  )

);
```

### GlobalContextsConfiguration

The `GlobalContextsConfiguration` class accepts the following:

| Attribute  | Type                   | Description                                              |
| ---------- | ---------------------- | -------------------------------------------------------- |
| `contexts` | `List<SelfDescribing>` | List of static context entities to attach to all events. |

## Add and remove global context at runtime

You can add and remove global context dynamically at any time after tracker initialization. Each global context entity is identified by a string `tag`, which lets you manage multiple entities independently.

### Add a global context entity

Use `addGlobalContexts()` with a tag and a context entity:

```dart
const myContext = SelfDescribing(

  schema: 'iglu:com.my_company/feature_flags/jsonschema/1-0-0',

  data: {

    'feature_a': true,

    'feature_b': false

  }

);



// Add using the tracker instance

await tracker.addGlobalContexts('feature_flags_tag', myContext);



// Or using the static method (requires specifying the tracker namespace)

await Snowplow.addGlobalContexts(

  'feature_flags_tag',

  myContext,

  tracker: 'ns1'

);
```

### Remove a global context entity

Use `removeGlobalContexts()` with the tag of the context entity you want to remove:

```dart
// Remove using the tracker instance

await tracker.removeGlobalContexts('feature_flags_tag');



// Or using the static method

await Snowplow.removeGlobalContexts(

  'feature_flags_tag',

  tracker: 'ns1'

);
```

> **Note:** Global context was added in 0.9.0. In this release, only static context entities are supported.

---

# Add entities and subject data to Flutter tracker events
> Enrich Flutter events with custom context entities using self-describing JSON schemas. Configure subject information including user ID, platform data, timezone, and screen resolution.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/adding-data/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset:

1. Event context using self-describing data: attach event context, describing anything you like, in the form of self-describing JSONs.
2. Global context: attach context entities to all events globally, without specifying them on each individual call.
3. Subject: include information about the user, or the platform on which the event occurred.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event’s entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`context`) in your data storage. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

```cpp
tracker.track(

    Structured(category: 'shop', action: 'add-to-basket'),

    contexts: [

        const SelfDescribing(

            schema: 'iglu:com.my_company/movie_poster/jsonschema/1-0-0',

            data: {

                'movie_name': 'Solaris',

                'poster_country': 'JP',

                'poster_date': '1978-01-01'

            }

        ),

        const SelfDescribing(

            schema: 'iglu:com.my_company/customer/jsonschema/1-0-0',

            data: {

                'p_buy': 0.23,

                'segment': 'young_adult'

            }

        )

    ]

);
```

## Global context

Global context entities are attached to all events without requiring you to pass them to each individual `track()` call. This is useful for data that applies across all or most of your tracked events.

See the [Global context page](/docs/sources/flutter-tracker/adding-data/global-context/) for detailed documentation on configuring global contexts at initialization time and adding or removing them dynamically.

## Adding user and platform data with Subject

Subject information describes the user and device associated with the event, such as their user ID, what type of device they used, or what size screen that device had.

This information can be entered during tracker initialization by passing a `SubjectConfiguration` instance to the `Snowplow.createTracker` method. All of the information is optional.

Some subject information is filled automatically by the tracker. This includes the platform of the user, timezone, language, resolution, and viewport.

Please refer to the section on subject configuration on the [Configuration page](/docs/sources/flutter-tracker/initialization-and-configuration/) to learn more.

---

# Enable anonymous tracking with the Flutter tracker
> Anonymize user identifiers including domain_userid, session IDs, network_userid, and IP addresses. Configure client-side and server-side anonymization using userAnonymisation and serverAnonymisation flags.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/anonymous-tracking/

Anonymous tracking is a feature that enables anonymization of various user and session identifiers, to support user privacy when consent for tracking the identifiers isn't given.

This table shows the [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) that can be anonymized on web and mobile:

| Identifier          | Location in event        | Tracked on web                              | Tracked on mobile  |
| ------------------- | ------------------------ | ------------------------------------------- | ------------------ |
| `user_id`           | Atomic                   | ✅                                           | ✅                  |
| `domain_userid`     | Atomic                   | ✅ Non-configurable                          | ✅                  |
| `domain_sessionid`  | Atomic                   | ✅ Non-configurable                          | ❌                  |
| `domain_sessionidx` | Atomic                   | ✅ Non-configurable                          | ❌                  |
| `userId`            | Session entity           | ✅ Non-configurable, same as `domain_userid` | ✅ Non-configurable |
| `sessionId`         | Session entity           | ✅ Non-configurable                          | ✅ Non-configurable |
| `previousSessionId` | Session entity           | ✅ Non-configurable                          | ✅ Non-configurable |
| `appleIdfa`         | Mobile (platform) entity | ❌                                           | ✅                  |
| `appleIdfv`         | Mobile (platform) entity | ❌                                           | ✅                  |
| `androidIdfa`       | Mobile (platform) entity | ❌                                           | ✅                  |
| `network_userid`    | Atomic                   | ✅ Non-configurable                          | ✅                  |
| `user_ipaddress`    | Atomic                   | ✅ Non-configurable                          | ✅                  |

You can set some of these properties at [tracker initialization](/docs/sources/flutter-tracker/initialization-and-configuration/).

> **Note:** The Collector captures the IP address from the request HTTP headers, and updates the `user_ipaddress` event property. However, if you set the `user_ipaddress` property at initialization, that value has priority.
>
> Similarly, if you set the `network_userid` property, that value is used instead of the Collector cookie value.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

> **Note:** As well as setting these flags at initialization, from version 0.10.0 you can change them at runtime on an existing tracker, without recreating it. See [Changing anonymisation at runtime](#changing-anonymisation-at-runtime) below.

There are several levels to the anonymization depending on which of the categories are affected.

## 1. Full client-side anonymization

In this case, we want to anonymize both the client-side user identifiers as well as the client-side session identifiers. Here the Session entity is not configured: no session information will be tracked at all.

```dart
const TrackerConfiguration()

    .userAnonymisation(true)

    .sessionContext(false)

    .platformContext(true) // only relevant for mobile
```

On web, setting `userAnonymisation` stops any user identifiers or session information being stored in cookies or localStorage. This means that `domain_userid`, `domain_sessionid`, and `domain_sessionidx` will be anonymized. These properties are already not present in any mobile events.

On mobile, setting `userAnonymisation` affects properties in the Session and Platform entities. In this example, the Platform entity is enabled. The `appleIdfv` Platform property would be anonymized, as well as `appleIdfa`/`androidIdfa` if your app is configured to track those.

**Web:**

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `domain_sessionid`  | Atomic                   | ❌                   |
| `domain_sessionidx` | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | N/A                 |
| `appleIdfv`         | Mobile (platform) entity | N/A                 |
| `androidIdfa`       | Mobile (platform) entity | N/A                 |
| `network_userid`    | Atomic                   | ✅                   |
| `user_ipaddress`    | Atomic                   | ✅                   |

**Mobile:**

| Identifier          | Location in event        | Included in event?          |
| ------------------- | ------------------------ | --------------------------- |
| `user_id`           | Atomic                   | ❌                           |
| `domain_userid`     | Atomic                   | ❌                           |
| `domain_sessionid`  | Atomic                   | N/A                         |
| `domain_sessionidx` | Atomic                   | N/A                         |
| `userId`            | Session entity           | ❌ no session entity         |
| `sessionId`         | Session entity           | ❌ no session entity         |
| `previousSessionId` | Session entity           | ❌ no session entity         |
| `appleIdfa`         | Mobile (platform) entity | ❌                           |
| `appleIdfv`         | Mobile (platform) entity | ❌                           |
| `androidIdfa`       | Mobile (platform) entity | ❌                           |
| `network_userid`    | Atomic                   | ✅/❌ removed if you set this |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you set this |

***

## 2. Client-side anonymization with session tracking

This setting disables client-side user identifiers but tracks session information.

```dart
const TrackerConfiguration()

    .userAnonymisation(true)

    .sessionContext(true)

    .platformContext(true) // only relevant for mobile
```

Enabling both `userAnonymisation` and `sessionContext` means that events will have the Session entity, but the `userId` property will be anonymised to a null UUID (`00000000-0000-0000-0000-000000000000`). The `sessionId` and `sessionIndex` are still present, as are `domain_sessionid` and `domain_sessionidx` for web events. As above, if the Platform entity is enabled, the IDFA identifiers will not be present.

**Web:**

| Identifier          | Location in event        | Included in event? |
| ------------------- | ------------------------ | ------------------ |
| `user_id`           | Atomic                   | ❌                  |
| `domain_userid`     | Atomic                   | ❌                  |
| `domain_sessionid`  | Atomic                   | ✅                  |
| `domain_sessionidx` | Atomic                   | ✅                  |
| `userId`            | Session entity           | ❌ null UUID        |
| `sessionId`         | Session entity           | ✅                  |
| `previousSessionId` | Session entity           | ❌                  |
| `appleIdfa`         | Mobile (platform) entity | N/A                |
| `appleIdfv`         | Mobile (platform) entity | N/A                |
| `androidIdfa`       | Mobile (platform) entity | N/A                |
| `network_userid`    | Atomic                   | ✅                  |
| `user_ipaddress`    | Atomic                   | ✅                  |

**Mobile:**

| Identifier          | Location in event        | Included in event?          |
| ------------------- | ------------------------ | --------------------------- |
| `user_id`           | Atomic                   | ❌                           |
| `domain_userid`     | Atomic                   | ❌                           |
| `domain_sessionid`  | Atomic                   | N/A                         |
| `domain_sessionidx` | Atomic                   | N/A                         |
| `userId`            | Session entity           | ❌ null UUID                 |
| `sessionId`         | Session entity           | ✅                           |
| `previousSessionId` | Session entity           | ❌                           |
| `appleIdfa`         | Mobile (platform) entity | ❌                           |
| `appleIdfv`         | Mobile (platform) entity | ❌                           |
| `androidIdfa`       | Mobile (platform) entity | ❌                           |
| `network_userid`    | Atomic                   | ✅/❌ removed if you set this |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you set this |

***

## 3. Server-side anonymization

Server-side anonymization affects user identifiers set server-side. In particular, these are the `network_userid` property set in server-side cookie and the user IP address (`user_ipaddress`). You can anonymize the properties using the `serverAnonymisation` flag in `EmitterConfiguration`:

```dart
const EmitterConfiguration()

    .serverAnonymisation(true)
```

Setting this will add a `SP-Anonymous` HTTP header to requests sent to the Snowplow collector. The Snowplow pipeline will anonymize the identifiers.

On mobile, `serverAnonymisation` and `userAnonymisation` are separate. This means you can anonymize only the server-side identifiers without also anonymizing the client-side identifiers.

On web, setting `serverAnonymisation` provides full anonymization. It's effectively a subset of `userAnonymisation`. If `serverAnonymisation` is enabled, this will also set `userAnonymisation` to `true` (even if set to `false` in your `TrackerConfiguration`).

**Web:**

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `domain_sessionid`  | Atomic                   | ❌                   |
| `domain_sessionidx` | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | N/A                 |
| `appleIdfv`         | Mobile (platform) entity | N/A                 |
| `androidIdfa`       | Mobile (platform) entity | N/A                 |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

**Mobile:**

| Identifier          | Location in event        | Included in event?                |
| ------------------- | ------------------------ | --------------------------------- |
| `user_id`           | Atomic                   | ✅                                 |
| `domain_userid`     | Atomic                   | ✅                                 |
| `domain_sessionid`  | Atomic                   | N/A                               |
| `domain_sessionidx` | Atomic                   | N/A                               |
| `userId`            | Session entity           | ✅                                 |
| `sessionId`         | Session entity           | ✅                                 |
| `previousSessionId` | Session entity           | ✅                                 |
| `appleIdfa`         | Mobile (platform) entity | ✅                                 |
| `appleIdfv`         | Mobile (platform) entity | ✅                                 |
| `androidIdfa`       | Mobile (platform) entity | ✅                                 |
| `network_userid`    | Atomic                   | ❌/✅ still present if you set this |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you set this |

***

## 4. Full anonymization for mobile

For full anonymization on mobile, set both client-side and server-side anonymization.

```dart
const TrackerConfiguration()

    .userAnonymisation(true)

    .sessionContext(false)

    .platformContext(true) // only relevant for mobile



const EmitterConfiguration()

    .serverAnonymisation(true)
```

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `domain_sessionid`  | Atomic                   | N/A                 |
| `domain_sessionidx` | Atomic                   | N/A                 |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌                   |
| `appleIdfv`         | Mobile (platform) entity | ❌                   |
| `androidIdfa`       | Mobile (platform) entity | ❌                   |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

## Changing anonymisation at runtime

> **Note:** This feature is only available on mobile (Android and iOS).

The `userAnonymisation` and `serverAnonymisation` flags described above are normally set at [tracker initialization](/docs/sources/flutter-tracker/initialization-and-configuration/). You can also change them at runtime on an existing tracker, for example when a user updates their tracking consent, without recreating the tracker:

```dart
// On the tracker instance:

await tracker.setUserAnonymisation(true);

await tracker.setServerAnonymisation(true);



// Or via the static API, using the tracker namespace:

await Snowplow.setUserAnonymisation(true, tracker: 'ns1');

await Snowplow.setServerAnonymisation(true, tracker: 'ns1');
```

This avoids recreating the tracker (and therefore resetting identifiers) just to change the anonymisation level. The new setting applies to all events tracked afterwards, with the same effect as setting the flag at initialization.

> **Note:** Changing `userAnonymisation` starts a new session. The tracker instance and the user identifier are preserved, but the `sessionId` changes. `setServerAnonymisation` does not affect the session.

These methods are only available on mobile (Android and iOS). On Web they throw an `Unimplemented` `PlatformException`.

---

# Get started with the Flutter tracker
> Install the Snowplow Flutter tracker from pub.dev and initialize tracking in your Flutter app. Create a tracker instance with namespace and collector endpoint to track events across Web, Android, and iOS platforms.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/getting-started/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

The following steps will guide you through setting up the Flutter tracker in your project and tracking a simple event.

## Installation

Add the Snowplow tracker as a dependency to your Flutter application:

```bash
flutter pub add snowplow_tracker
```

This will add a line with the dependency to your pubspec.yaml:

```yaml
dependencies:

  snowplow_tracker: ^0.8.0
```

Import the package into your Dart code:

```dart
import 'package:snowplow_tracker/snowplow_tracker.dart'
```

### Installation on Web

If using the tracker within a Flutter app for Web, you will also need to import the Snowplow JavaScript Tracker in your `index.html` file. Please load the JS tracker with the Snowplow tag as [described here](/docs/sources/web-trackers/tracker-setup/). Do not change the global function name `snowplow` that is used to access the tracker – the Flutter APIs assume that it remains the default as shown in documentation.

Make sure to use JavaScript tracker version `3.5` or newer. You may also refer to the [example project](https://github.com/snowplow/snowplow-flutter-tracker/tree/main/example) in the Flutter tracker repository to see this in action.

## Initialization

Instantiate a tracker using the `Snowplow.createTracker` function. At its most basic, the function takes two required arguments: `namespace` and `endpoint`. Tracker namespace identifies the tracker instance, you may create multiple trackers with different namespaces. The endpoint is the URI of the Snowplow collector to send the events to. This tracker creation is asynchronous and uses the `await` keyword; therefore it must occur inside a function labelled `async`. You could create the tracker in the `main()` of your main widget.

```dart
SnowplowTracker tracker = await Snowplow.createTracker(

    namespace: 'ns1',

    endpoint: 'http://...'

);



// Creating a tracker in the main() function

// It is passed to the other widgets as necessary

Future<void> main() async {

  WidgetsFlutterBinding.ensureInitialized();



  SnowplowTracker tracker = await Snowplow.createTracker(

      namespace: "namespace",

      endpoint: "http://0.0.0.0:9090");



  runApp(MyApp(tracker: tracker));

}
```

There are additional optional arguments to configure the tracker. To learn more about configuring how events are sent, check out [this page](/docs/sources/flutter-tracker/initialization-and-configuration/).

## Tracking events

To track events, simply instantiate their respective types (e.g., `ScreenView`, `SelfDescribing`, `Structured`) and pass them to the `tracker.track` or `Snowplow.track` methods.

```dart
tracker.track(ScreenView(

    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',

    name: 'home',

    type: 'full',

    transitionType: 'none'));
```

Visit documentation about [tracking events](/docs/sources/flutter-tracker/tracking-events/) to learn about other supported event types. You may also want to read about [adding more data to tracked events](/docs/sources/flutter-tracker/adding-data/).

## Testing

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

---

# Track events from hybrid Flutter apps with WebViews
> Enable event tracking from WebView content in Flutter apps using webview_flutter package. Forward Web tracker events to Flutter tracker or use WebView tracker directly to share sessions.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/hybrid-apps/

> **Info:** This feature is available since v0.8 of the Flutter tracker and supports WebView implemented using the [`webview_flutter` package](https://pub.dev/packages/webview_flutter).

Hybrid apps are mobile apps that in addition to a Flutter interface, provide part of the UI through an embedded Web view. Snowplow events are tracked from both the Flutter code as well as the Web view. Our goal is to have both events tracked from both places to share the same session and appear as tracked with the same tracker.

## Event forwarding

We recommend using the Web tracker (v4.3+) to forward all Web events to the Flutter tracker.

1. Implement the Flutter tracker.

2. Implement the [Snowplow Web/JavaScript tracker](/docs/sources/web-trackers/) in the WebView in your app. Make sure to include the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/).

3. Subscribe to WebView event messages. Here is a simplified example usage:

   ```dart
   import 'package:flutter/services.dart';

   import 'package:snowplow_tracker/snowplow_tracker.dart';



   class MainPage extends StatefulWidget {

       const MainPage({super.key, required this.tracker});



       // Snowplow tracker instance

       final SnowplowTracker tracker;



       @override

       State<MainPage> createState() => _MainPageState();

   }



   class _MainPageState extends State<MainPage> with WidgetsBindingObserver {

       late final WebViewController _webViewController;



       @override

       void initState() {

           _webViewController = WebViewController()

               ..setJavaScriptMode(JavaScriptMode.unrestricted)

               ..loadRequest(Uri.parse(const String.fromEnvironment('WEBVIEW_URL',

                   defaultValue: 'http://localhost:3000')));



           // register the webview controller with the tracker to set up JavaScript channel

           widget.tracker.registerWebViewJavaScriptChannel(

               webViewController: _webViewController);

       }



       @override

       Widget build(BuildContext context) {

           return SizedBox(

               height: 300,

               child: WebViewWidget(controller: _webViewController),

           );

       }

   }
   ```

4. Track events as usual.

The Web tracker will automatically intercept all web events and forward them to the Flutter tracker. The forwarded events will have the tracker version from Web, e.g. "js-4.1.0", but will otherwise be tracked like the mobile events. They may contain additional information not present in the Flutter mobile events, such as a browser useragent string or URL, or Web context entities e.g. the [WebPage entity](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity).

The forwarded events are filtered out of the Web tracker event queue so that they are not tracked twice.

The WebView plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## WebView Tracker

If you don't want to implement a Web tracker in your WebView, you can use the [Snowplow WebView tracker](/docs/sources/webview-tracker/) directly. This could be suitable if you only want to track a small number of events from the WebView. We recommend event forwarding for most use cases.

1. Implement the Flutter tracker.
2. Implement the [Snowplow WebView tracker](/docs/sources/webview-tracker/) in the WebView in your app.
3. Subscribe to WebView event messages. This step is the same as shown above.
4. Manually track [WebView events](/docs/sources/webview-tracker/).

All event types can be tracked with WebView tracker v0.3.0+, but it requires more work than using the event forwarding.

---

# Flutter tracker SDK
> The Snowplow Flutter Tracker enables cross-platform event tracking from Flutter applications across Web, Android, and iOS. Track screen views, custom events, and user sessions with unified data model support.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/

The Snowplow Flutter Tracker allows you to add analytics to your Flutter apps for Web, Android, and iOS when using a Snowplow pipeline.

The tracker is published on pub.dev as [snowplow\_tracker](https://pub.dev/packages/snowplow_tracker).

---

# Initialize and configure the Flutter tracker
> Initialize the Flutter tracker with createTracker method and configure TrackerConfiguration, SubjectConfiguration, GdprConfiguration, and EmitterConfiguration. Set up app identifiers, context entities, session tracking, and anonymous tracking options.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/initialization-and-configuration/

The package provides a single method to initialize and configure a new tracker, the `Snowplow.createTracker` method. It accepts configuration parameters for the tracker and returns a `SnowplowTracker` instance.

```dart
SnowplowTracker tracker = await Snowplow.createTracker(

    namespace: 'ns1',

    endpoint: 'http://...',

    trackerConfig: const TrackerConfiguration(...),

    gdprConfig: const GdprConfiguration(...),

    subjectConfig: const SubjectConfiguration(...));

);
```

The method returns a `SnowplowTracker` instance. This can be later used for tracking events, or accessing tracker properties. However, all methods provided by the `SnowplowTracker` instance are also available as static functions in the `Snowplow` class but they require passing the tracker namespace as string.

The only required attributes of the `Snowplow.createTracker` method are `namespace` used to identify the tracker, and the Snowplow collector `endpoint`. Additionally, one can configure the HTTP method to be used when sending events to the collector, as well as a custom POST path, and provide configuration by instantiating classes for `TrackerConfiguration`, `SubjectConfiguration`, or `GdprConfiguration`. By default, events are sent by POST. The following arguments are accepted by the `Snowplow.createTracker` method:

| Attribute              | Type                           | Description                                                                          |
| ---------------------- | ------------------------------ | ------------------------------------------------------------------------------------ |
| `namespace`            | `String`                       | Tracker namespace to identify the tracker.                                           |
| `endpoint`             | `String`                       | URI for the Snowplow collector endpoint.                                             |
| `method`               | `Method?`                      | HTTP method to use: `Method.get` or `Method.post` (`Method.post` is default).        |
| `customPostPath`       | `String?`                      | Custom POST path.                                                                    |
| `requestHeaders`       | `Map<String, String>?`         | Map of custom HTTP headers to add to requests to the collector.                      |
| `trackerConfig`        | `TrackerConfiguration?`        | Configuration of the tracker and the core tracker properties.                        |
| `gdprConfig`           | `GdprConfiguration?`           | Determines the GDPR context that will be attached to all events sent by the tracker. |
| `subjectConfig`        | `SubjectConfiguration?`        | Subject information about tracked user and device that is added to events.           |
| `emitterConfig`        | `EmitterConfiguration?`        | Configuration for how the events are sent.                                           |
| `globalContextsConfig` | `GlobalContextsConfiguration?` | Configuration of global context entities attached to all events.                     |

> **Note:** The ability to set `customPostPath` was added in v0.2.0. Setting a custom POST path can be useful in avoiding adblockers; it replaces the default "com.snowplowanalytics/snowplow/tp2". Your event collector must also be configured to accept the custom path.

> **Note:** The `EmitterConfiguration` class was added in v0.3.0.

> **Note:** The `GlobalContextsConfiguration` class was added in 0.9.0.

## Configuration of tracker properties: `TrackerConfiguration`

`TrackerConfiguration` provides options to configure properties and features of the tracker. In addition to setting the app identifier and device platform, the configuration enables turning several automatic context entities on and off.

| Attribute                      | Type                         | Description                                                                                                                                                                                | Android | iOS | Web | Default                                       |
| ------------------------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | --- | --- | --------------------------------------------- |
| `appId`                        | `String?`                    | Identifier of the app.                                                                                                                                                                     | ✔       | ✔   | ✔   | null on Web, bundle identifier on iOS/Android |
| `devicePlatform`               | `DevicePlatform?`            | The device platform the tracker runs on. Available options are provided by the `DevicePlatform` enum.                                                                                      | ✔       | ✔   | ✔   | "web" on Web, "mob" on iOS/Android            |
| `base64Encoding`               | `bool?`                      | Indicates whether payload JSON data should be base64 encoded.                                                                                                                              | ✔       | ✔   | ✔   | true                                          |
| `platformContext`              | `bool?`                      | Indicates whether [platform](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-2) (mobile) entity should be attached to tracked events.          | ✔       | ✔   |     | true                                          |
| `geoLocationContext`           | `bool?`                      | Indicates whether [geo-location](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-1-0) entity should be attached to tracked events.          | ✔       | ✔   | ✔   | false                                         |
| `sessionContext`               | `bool?`                      | Indicates whether [session](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) entity should be attached to tracked events.                    | ✔       | ✔   | ✔   | true                                          |
| `webPageContext`               | `bool?`                      | Indicates whether a context entity about current [web page](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0) should be attached to tracked events. |         |     | ✔   | true                                          |
| `screenContext`                | `bool?`                      | Indicates whether [screen](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/screen/jsonschema/1-0-0) entity should be attached to tracked events.                               | ✔       | ✔   |     | true                                          |
| `applicationContext`           | `bool?`                      | Indicates whether [application](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/application/jsonschema/1-0-0) entity should be attached to tracked events.                     | ✔       | ✔   |     | true                                          |
| `webActivityTracking`          | `WebActivityTracking?`       | Enables activity tracking using page views and pings on the Web.                                                                                                                           |         |     | ✔   | true                                          |
| `userAnonymisation`            | `bool?`                      | Anonymizes certain user identifiers.                                                                                                                                                       | ✔       | ✔   | ✔   | false                                         |
| `lifecycleAutotracking`        | `bool?`                      | Indicates whether the [lifecycle](iglu:com.snowplowanalytics.mobile/application_lifecycle/jsonschema/1-0-0) entity and foreground and background events should be autotracked.             | ✔       | ✔   |     | true                                          |
| `screenEngagementAutotracking` | `bool?`                      | Indicates whether to enable tracking of the screen end event and the screen summary context entity.                                                                                        | ✔       | ✔   |     | true                                          |
| `platformContextProperties`    | `PlatformContextProperties?` | Overrides for the values for properties of the platform context entity.                                                                                                                    | ✔       | ✔   |     | null                                          |

> **Note:** The ability to enable `userAnonymisation`, or the screen and application context entities, was added in v0.3.0.

> **Note:** The ability to enable `lifecycleAutotracking` was added in v0.5.0.

The optional `WebActivityTracking` property configures page tracking on Web. Initializing the configuration will inform `SnowplowObserver` observers (see section on auto-tracking in "Tracking events") to auto track `PageViewEvent` events instead of `ScreenView` events on navigation changes. Further, setting the `minimumVisitLength` and `heartbeatDelay` properties of the `WebActivityTracking` instance will enable activity tracking using 'page ping' events on Web.

Activity tracking monitors whether a user continues to engage with a page over time, and record how they digest content on the page over time. That is accomplished using 'page ping' events. If activity tracking is enabled, the web page is monitored to see if a user is engaging with it. (E.g. is the tab in focus, does the mouse move over the page, does the user scroll etc.) If any of these things occur in a set period of time (`minimumVisitLength` seconds from page load and every `heartbeatDelay` seconds after that), a page ping event fires, and records the maximum scroll left / right and up / down in the last ping period. If there is no activity in the page (e.g. because the user is on a different browser tab), no page ping fires.

Lifecycle autotracking is only available on mobile apps (iOS and Android). When configured (it is enabled by default), a Lifecycle context entity is attached to all events. It records whether the app was visible or not when the event was tracked. In addition, a `Background` event will be tracked when the app is moved to background, and a `Foreground` event when the app moves back to foreground (becomes visible on the screen).

Screen engagement autotracking is also only available on mobile apps (iOS and Android). When configured (it is enabled by default), a screen summary context entity will be tracked along with screen end, foreground and background events. Make sure that you have lifecycle autotracking enabled for screen summary to have complete information.

See [this page](/docs/sources/flutter-tracker/anonymous-tracking/) for information about anonymous tracking.

## Configuration of emitter properties: `EmitterConfiguration`

This Configuration class was added in v0.3.0. Currently, the only property is `serverAnonymisation`.

| Attribute             | Type    | Description                                        | Android | iOS | Web | Default |
| --------------------- | ------- | -------------------------------------------------- | ------- | --- | --- | ------- |
| `serverAnonymisation` | `bool?` | Prevents tracking of server-side user identifiers. | ✔       | ✔   | ✔   | false   |

## Configuration of subject information: `SubjectConfiguration`

Subject information are persistent and global information about the tracked device or user. They apply to all events and are assigned as event properties.

Some of the properties are only configurable on iOS and Android and are automatically assigned on the Web.

| Attribute          | Type      | Description                                                     | Android | iOS | Web                              | Default |
| ------------------ | --------- | --------------------------------------------------------------- | ------- | --- | -------------------------------- | ------- |
| `userId`           | `String?` | Business ID of the user.                                        | ✔       | ✔   | ✔                                |         |
| `networkUserId`    | `String?` | Network user ID (UUIDv4).                                       | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `domainUserId`     | `String?` | Domain user ID (UUIDv4).                                        | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `userAgent`        | `String?` | Custom user-agent. It overrides the user-agent used by default. | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `ipAddress`        | `String?` | IP address.                                                     | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `timezone`         | `String?` | The timezone label.                                             | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `language`         | `String?` | The language set on the device.                                 | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `screenResolution` | `Size?`   | The screen resolution on the device.                            | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `screenViewport`   | `Size?`   | The screen viewport.                                            | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `colorDepth`       | `double?` | The color depth.                                                | ✔       | ✔   | Non-configurable, auto-assigned. |         |

The configured attributes are mapped to Snowplow event properties described in the [Snowplow Tracker Protocol](/docs/events/). They are mapped as follows:

| Attribute                 | Event Property      |
| ------------------------- | ------------------- |
| `userId`                  | `uid`               |
| `networkUserId`           | `network_userid`    |
| `domainUserId`            | `domain_userid`     |
| `userAgent`               | `useragent`         |
| `ipAddress`               | `user_ipaddress`    |
| `timezone`                | `os_timezone`       |
| `language`                | `lang`              |
| `screenResolution.width`  | `dvce_screenwidth`  |
| `screenResolution.height` | `dvce_screenheight` |
| `screenViewport.width`    | `br_viewwidth`      |
| `screenViewport.height`   | `br_viewheight`     |
| `colorDepth`              | `br_colordepth`     |

## GDPR context entity configuration: `GdprConfiguration`

Determines the GDPR context that will be attached to all events sent by the tracker.

| Attribute             | Type     | Description                  | Android | iOS | Web | Default |
| --------------------- | -------- | ---------------------------- | ------- | --- | --- | ------- |
| `basisForProcessing`  | `String` | Basis for processing.        | ✔       | ✔   | ✔   |         |
| `documentId`          | `String` | ID of a GDPR basis document. | ✔       | ✔   | ✔   |         |
| `documentVersion`     | `String` | Version of the document.     | ✔       | ✔   | ✔   |         |
| `documentDescription` | `String` | Description of the document. | ✔       | ✔   | ✔   |         |

## Global context configuration: `GlobalContextsConfiguration`

Configures global context entities to attach to all events. Unlike per-event context entities, global context entities are sent with every event without needing to specify them on individual `track()` calls.

| Attribute  | Type                   | Description                                               | Android | iOS | Web | Default |
| ---------- | ---------------------- | --------------------------------------------------------- | ------- | --- | --- | ------- |
| `contexts` | `List<SelfDescribing>` | List of context entities to attach to all tracked events. | ✔       | ✔   | ✔   | empty   |

See the [Global context page](/docs/sources/flutter-tracker/adding-data/global-context/) for detailed information on configuring global contexts at initialization time and managing them dynamically at runtime.

---

# Configure session tracking with the Flutter tracker and Unified Digital package
> Enable session tracking with client_session context entity across Android, iOS, and Web platforms. Configure foreground and background timeouts to track user sessions consistently.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/sessions-and-data-model/

The Flutter tracker gives you the option to adopt the [Snowplow Unified data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) across all supported platforms – Android, iOS, and Web.

In addition to adopting screen view events, the unified data model defines that sessions are represented using a [context entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-1) where it exists. Concretely, the `client_session` context entity is added to all tracked events if session tracking is enabled in the tracker configuration (through the `sessionContext` property). This entity consists of the following properties:

| Attribute           | Description                                                               | Required? |
| ------------------- | ------------------------------------------------------------------------- | --------- |
| `userId`            | An identifier for the user of the session.                                | Yes       |
| `sessionId`         | An identifier (UUID) for the session.                                     | Yes       |
| `sessionIndex`      | The index of the current session for this user.                           | Yes       |
| `previousSessionId` | The previous session identifier (UUID) for this user.                     | No        |
| `storageMechanism`  | The mechanism that the session information has been stored on the device. | Yes       |
| `firstEventId`      | The optional identifier (UUID) of the first event id for this session.    | No        |

Behind the scenes, the Flutter tracker uses the default configuration for session management on the Android, iOS, and Web trackers.

On Android and iOS, session data is maintained for the life of the application being installed on a device. Essentially it will update if it is not accessed within a configurable timeout. There are two inactivity timeouts that result in updates to the `sessionId`: foreground inactivity, and background inactivity timeout. The default 30 minutes setting is used for both.

On the Web, the tracker uses domain (`duid`) and session cookies (`sid`) as implemented by the JavaScript tracker. The session cookie expires after 30 minutes of inactivity. This means that a user leaving the site and returning in under 30 minutes does not change the session. In contrast with the JavaScript tracker, the Flutter tracker also adds the `client_session` context entity that wraps the domain and session IDs.

---

# Track events with the Flutter tracker
> Track events in Flutter using SelfDescribing, Structured, ScreenView, Timing, and Consent events. Enable auto-tracking of page views with SnowplowObserver or manually track custom events with schemas.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/tracking-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in event classes to help you track different kinds of events. When instantiated, their objects can be passed to the `Snowplow.track()` method to send events to the Snowplow collector. The event classes range from single purpose ones, such as `ScreenView`, to the more complex but flexible `SelfDescribing`, which can be used to track any kind of user behavior. We strongly recommend using `SelfDescribing` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

## Auto-tracked view events

There is an option to automatically track view events when currently active pages change through the [Navigator API](https://api.flutter.dev/flutter/widgets/Navigator-class.html).

To activate this feature, one has to register a `SnowplowObserver` retrieved from the tracker instance using `SnowplowTracker.getObserver()`. The retrieved observer can be added to `navigatorObservers` in `MaterialApp`:

```dart
MaterialApp(

  navigatorObservers: [

    tracker.getObserver()

  ],

  ...

);
```

If using the `Router` API with the `MaterialApp.router` constructor, add the observer to the `observers` of your `Navigator` instance, e.g.:

```dart
Navigator(

  observers: [tracker.getObserver()],

  ...

);
```

The `SnowplowObserver` automatically tracks `PageViewEvent` and `ScreenView` events when the currently active `ModalRoute` of the navigator changes.

By default, `ScreenView` events are tracked on all platforms. In case `TrackerConfiguration.webActivityTracking` is configured when creating the tracker, `PageViewEvent` events will be tracked on Web instead of `ScreenView` events (`ScreenView` events will still be tracked on other platforms).

The `SnowplowTracker.getObserver()` function takes an optional `nameExtractor` function as argument which is used to extract a name from new routes that is used in tracked `ScreenView` or `PageViewEvent` events.

The following operations will result in tracking a view event:

```dart
Navigator.pushNamed(context, '/contact/123');



Navigator.push<void>(context, MaterialPageRoute(

  settings: RouteSettings(name: '/contact/123'),

  builder: (_) => ContactDetail(123)));



Navigator.pushReplacement<void>(context, MaterialPageRoute(

  settings: RouteSettings(name: '/contact/123'),

  builder: (_) => ContactDetail(123)));



Navigator.pop(context);
```

## Manually-tracked events

Event classes supported by the Flutter Tracker:

| Method             | Event type tracked                                          |
| ------------------ | ----------------------------------------------------------- |
| `SelfDescribing`   | Custom event based on "self-describing" JSON schema         |
| `Structured`       | Semi-custom structured event                                |
| `ScreenView`       | View of a screen in the app                                 |
| `Timing`           | User timing events such as how long resources take to load. |
| `ConsentGranted`   | User opting into data collection.                           |
| `ConsentWithdrawn` | User withdrawing consent for data collection.               |

All the methods share common features and parameters. Every type of event can have an optional context added. See the [next page](/docs/sources/flutter-tracker/adding-data/) to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event types, followed by the out-of-the-box event types. Note that you can also design and create your own page view, or screen view, using `selfDescribing`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

## Track self-describing events with `SelfDescribing`

Use the `SelfDescribing` type to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

Creating an instance of `SelfDescribing` takes a schema name and a dictionary of event data.

Example (assumes that `tracker` is a tracker instance created using `Snowplow.createTracker`):

```dart
tracker.track(SelfDescribing(

    schema: 'iglu:com.example_company/save_game/jsonschema/1-0-2',

    data: {

        'saveId': '4321',

        'level': 23,

        'difficultyLevel': 'HARD',

        'dlContent': true

    }

));
```

## Track structured events with `Structured`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category", "action", "label", and "value" properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| Argument   | Description                                                    | Required in event? |
| ---------- | -------------------------------------------------------------- | ------------------ |
| `category` | The grouping of structured events which this action belongs to | Yes                |
| `action`   | Defines the type of user interaction which this event involves | Yes                |
| `label`    | Often used to refer to the 'object' the action is performed on | No                 |
| `property` | Describing the 'object', or the action performed on it         | No                 |
| `value`    | Provides numerical data about the event                        | No                 |

Example:

```dart
tracker.track(Structured(

    category: 'shop',

    action: 'add-to-basket',

    label: 'Add To Basket',

    property: 'pcs',

    value: 2.00,

));
```

## Track page views with `PageViewEvent`

The `PageViewEvent` may be used to track page views on the Web. The event is designed to track web page views and automatically captures page title, referrer and URL. Being Web-only, it is not implemented on Android and iOS where the app is not displayed as a Web page.

Page view events are the basic building blocks for the [Snowplow Unified data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/).

## Track screen views with `ScreenView`

Use `ScreenView` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name`, `id`, `type`, and `transitionType`. The `name` and `id` properties are required. "Name" is the human-readable screen name, and "ID" should be the unique screen ID (UUID v4).

Screen view events are also used in the[Snowplow Unified data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/). Nevertheless, the Flutter tracker also implements them on Web. You may adopt them for use in the unified data model to provide consistent event tracking across all platforms.

This method creates an unstruct event, by creating and tracking a self-describing event. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the parameters which you provide. That schema is hosted on the schema repository Iglu Central, and so will always be available to your pipeline.

| Argument         | Description                                                 | Required in event? |
| ---------------- | ----------------------------------------------------------- | ------------------ |
| `name`           | The name of the screen viewed.                              | Yes                |
| `id`             | The id (UUID v4) of screen that was viewed.                 | Yes                |
| `type`           | The type of screen that was viewed.                         | No                 |
| `previousName`   | The name of the previous screen that was viewed.            | No                 |
| `previousType`   | The type of screen that was viewed.                         | No                 |
| `previousId`     | The id (UUID v4) of the previous screen that was viewed.    | No                 |
| `transitionType` | The type of transition that led to the screen being viewed. | No                 |

Example:

```dart
tracker.track(ScreenView(

    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',

    name: 'home',

    type: 'full',

    transitionType: 'none'));
```

### Screen engagement tracking

Screen engagement tracking is a feature that enables tracking the user activity on the screen. This consists of the time spent and the amount of content viewed on the screen.

Concretely, it consists of the following metrics:

1. Time spent on screen while the app was in foreground (tracked automatically).
2. Time spent on screen while the app was in background (tracked automatically).
3. Number of list items scrolled out of all list items (requires some manual tracking).
4. Scroll depth in pixels (requires some manual tracking).

This information is attached using a [`screen_summary` context entity](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) to the following events:

1. [`screen_end` event](/docs/events/ootb-data/page-activity-tracking/#screen-end-event) that is automatically tracked before a new screen view event.
2. [`application_background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event).
3. [`application_foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event).

Screen engagement tracking is enabled by default, but can be configured using the `TrackerConfiguration.screenEngagementAutotracking` option.

For a demo of how mobile screen engagement tracking works in action, **[please visit this demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/)**.

#### Updating list item view and scroll depth information

To update the list item viewed and scroll depth information tracked in the screen summary entity, you can track the `ListItemView` and `ScrollChanged` events with this information. When tracked, the tracker won't send these events individually to the collector, but will process the information into the next `screen_summary` entity and discard the events. You may want to track the events every time a new list item is viewed on the screen, or whenever the scroll position changes.

To update the list items viewed information:

```dart
Event event = const ListItemView(index: 1, itemsCount: 10);

tracker.track(event);
```

To update the scroll depth information:

```dart
Event event = const ScrollChanged(

  yOffset: 10,

  xOffset: 20,

  viewHeight: 100,

  viewWidth: 200,

  contentHeight: 300,

  contentWidth: 400,

);

tracker.track(event);
```

## Track timing events with `Timing`

Use the `Timing` type to track user timing events such as how long resources take to load. These events take a timing `category`, the `variable` being measured, and the `timing` time measurement. An optional `label` can be added to further identify the timing event

| Argument   | Description                                              | Required in event? |
| ---------- | -------------------------------------------------------- | ------------------ |
| `category` | Defines the timing category.                             | Yes                |
| `variable` | Defines the timing variable measured.                    | Yes                |
| `timing`   | Represents the time.                                     | Yes                |
| `label`    | An optional string to further identify the timing event. | No                 |

Example:

```dart
tracker.track(Timing(

    category: 'category',

    variable: 'variable',

    timing: 1,

    label: 'label',

));
```

## Track user consent with `ConsentGranted` and `ConsentWithdrawn`

Use the `ConsentGranted` to track a user opting into data collection and `ConsentWithdrawn` to track a user withdrawing their consent for data collection.

For both events, a consent document context will be attached to the event using the `documentId` and `version` arguments supplied. To specify that a user opts out of all data collection using the `ConsentWithdrawn` event, the `all` property should be set to true.

Properties of `ConsentGranted`:

| Argument              | Description                            | Required in event? |
| --------------------- | -------------------------------------- | ------------------ |
| `expiry`              | The expiry date-time of the consent.   | Yes                |
| `documentId`          | The consent document ID.               | Yes                |
| `version`             | The consent document version.          | Yes                |
| `name`                | Optional consent document name.        | No                 |
| `documentDescription` | Optional consent document description. | No                 |

Example:

```dart
tracker.track(ConsentGranted(

    expiry: DateTime.now(),

    documentId: '1234',

    version: '5',

    name: 'name1',

    documentDescription: 'description1',

));
```

Properties of `ConsentWithdrawn`:

| Argument              | Description                                   | Required in event? |
| --------------------- | --------------------------------------------- | ------------------ |
| `all`                 | Whether user opts out of all data collection. | Yes                |
| `documentId`          | The consent document ID.                      | Yes                |
| `version`             | The consent document version.                 | Yes                |
| `name`                | Optional consent document name.               | No                 |
| `documentDescription` | Optional consent document description.        | No                 |

Example:

```dart
tracker.track(ConsentWithdrawn(

    all: false,

    documentId: '1234',

    version: '5',

    name: 'name1',

    documentDescription: 'description1',

));
```

---

# Media tracking with the Flutter tracker
> Track media playback events in Flutter applications using the built-in media tracking capabilities available since version 0.7.0.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/tracking-events/media-tracking/

> **Note:** Media tracking is available in the Flutter tracker since version 0.7.0.

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

```dart
// 1. Start tracking when player loads

final id = const Uuid().v4();

MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: MediaTrackingConfiguration(

      id: id,

      player: const MediaPlayerEntity(

          duration: 300,

          label: 'My Video Title',

          mediaType: MediaType.video,

      )

  )

);



// 2. Update player state every second

await mediaTracking.update(

  player: MediaPlayerEntity(currentTime: currentTime)

);



// 3. Track events as they occur

await mediaTracking.track(MediaPlayEvent());

await mediaTracking.track(MediaPauseEvent());

await mediaTracking.track(MediaSeekStartEvent());

await mediaTracking.track(MediaSeekEndEvent());

await mediaTracking.track(MediaEndEvent());



// 4. End tracking when done

await tracker.endMediaTracking(id: id);
```

> **Info:** To track media events on web with the Flutter tracker, configure the JavaScript plugin URL:```dart
> SnowplowTracker tracker = await Snowplow.createTracker(
>
>   namespace: namespace,
>
>   endpoint: endpoint,
>
>   trackerConfig: const TrackerConfiguration(
>
>       jsMediaPluginURL: "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media@latest/dist/index.umd.min.js"
>
>   )
>
> );
> ```

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: const MediaTrackingConfiguration(

      id: id,

      player: MediaPlayerEntity(

          currentTime: 0,          // Current playback position in seconds

          duration: 300,           // Total duration in seconds

          ended: false,            // Whether playback has ended

          livestream: false,       // Whether this is a live stream

          label: 'My Video',       // Human-readable title (recommended)

          loop: false,             // Whether to restart after ending

          mediaType: MediaType.video, // MediaType.video or MediaType.audio

          muted: false,            // Whether audio is muted

          paused: true,            // Whether playback is paused

          pictureInPicture: false, // Whether in picture-in-picture mode

          playerType: 'html5',     // Player type identifier

          playbackRate: 1.0,       // Playback speed (1 = normal)

          quality: '1080p',        // Quality level

          volume: 100              // Volume percentage (0-100)

      )

  )

);
```

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: const MediaTrackingConfiguration(

      id: id,

      pingInterval: 30,   // Seconds between pings (default: 30)

      maxPausedPings: 1   // Max pings while paused (default: 1)

  )

);



// Or turn off pings entirely

MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: const MediaTrackingConfiguration(id: id, pings: false)

);
```

### Percentage progress

Track events when playback reaches specified percentage boundaries.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: const MediaTrackingConfiguration(

      id: id,

      boundaries: [10, 25, 50, 75, 90]  // Fire events at these percentages

  )

);
```

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: const MediaTrackingConfiguration(id: id, session: true)

);
```

### Custom entities

You can attach custom entities to all events for this media tracking instance.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(

  configuration: const MediaTrackingConfiguration(

      id: id,

      contexts: [

          SelfDescribing(

              schema: "iglu:com.example/video_metadata/jsonschema/1-0-0",

              data: {"contentId": "abc123", "category": "tutorial"}

          )

      ]

  )

);
```

### Filter events

Media event filtering isn't available for the Flutter tracker.

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

```dart
await mediaTracking.track(MediaReadyEvent());        // Player is ready

await mediaTracking.track(MediaPlayEvent());         // Playback started

await mediaTracking.track(MediaPauseEvent());        // Playback paused

await mediaTracking.track(MediaEndEvent());          // Playback ended

await mediaTracking.track(MediaSeekStartEvent());    // Seek operation started

await mediaTracking.track(MediaSeekEndEvent());      // Seek operation ended

await mediaTracking.track(MediaBufferStartEvent());  // Buffering started

await mediaTracking.track(MediaBufferEndEvent());    // Buffering ended
```

### Player state events

```dart
await mediaTracking.track(const MediaPlaybackRateChangeEvent(newRate: 1.5));

await mediaTracking.track(const MediaVolumeChangeEvent(newVolume: 80));

await mediaTracking.track(const MediaFullscreenChangeEvent(fullscreen: true));

await mediaTracking.track(const MediaPictureInPictureChangeEvent(pictureInPicture: true));

await mediaTracking.track(const MediaQualityChangeEvent(

  newQuality: '1080p',

  bitrate: 5000,

  framesPerSecond: 30,

  automatic: true

));

await mediaTracking.track(const MediaErrorEvent(

  errorCode: '500',

  errorName: 'NetworkError',

  errorDescription: 'Failed to load media'

));
```

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

```dart
// Ad break events

await mediaTracking.track(

  MediaAdBreakStartEvent(),

  adBreak: const MediaAdBreakEntity(

      breakId: 'break-1',

      name: 'pre-roll',

      breakType: MediaAdBreakType.linear,

      podSize: 3

  )

);

await mediaTracking.track(MediaAdBreakEndEvent());



// Ad events

await mediaTracking.track(

  MediaAdStartEvent(),

  ad: const MediaAdEntity(

      adId: 'ad-1',

      name: 'Product Ad',

      creativeId: 'creative-1',

      duration: 30,

      skippable: true

  )

);

await mediaTracking.track(MediaAdFirstQuartileEvent());

await mediaTracking.track(MediaAdMidpointEvent());

await mediaTracking.track(MediaAdThirdQuartileEvent());

await mediaTracking.track(MediaAdCompleteEvent());

await mediaTracking.track(MediaAdSkipEvent());

await mediaTracking.track(MediaAdClickEvent());

await mediaTracking.track(MediaAdPauseEvent());

await mediaTracking.track(MediaAdResumeEvent());
```

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

```dart
await mediaTracking.track(

  const SelfDescribing(

      schema: "iglu:com.example/video_interaction/jsonschema/1-0-0",

      data: {"action": "share", "platform": "twitter"}

  )

);
```

## Update player state

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```dart
await mediaTracking.update(

  player: MediaPlayerEntity(currentTime: currentTime)

);
```

---

# Track the Subject with the Golang tracker
> Attach user and device information to Golang tracker events using the Subject class. Set user ID, screen resolution, viewport, timezone, language, IP address, and user-agent for enriched event data.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/adding-extra-data-the-subject-class/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

Create a subject like this:

```go
subject := sp.InitSubject()
```

The Subject class has a set of `Set...()` methods to attach extra data relating to the user to all tracked events:

- `SetUserId`
- `SetScreenResolution`
- `SetViewport`
- `SetColorDepth`
- `SetTimezone`
- `SetLanguage`
- `SetIpAddress`
- `SetUseragent`
- `SetDomainUserId`
- `SetNetworkUserId`

### Tracking the Subject

The `Subject` can be used into two ways.

When initialising the `Tracker`:

```go
tracker := sp.InitTracker(

  sp.RequireEmitter(emitter),

  sp.OptionSubject(subject)

)
```

Or when you track an event:

```go
tracker.TrackPageView(sp.PageViewEvent{

  PageUrl: sphelp.NewString("acme.com"),

  Subject: subject

})
```

When setting the `Subject` as you track as event, you will override the `Tracker` level `Subject` for that specific event.

### Set user ID with `SetUserId`

You can set the user ID to any string:

```go
s.SetUserId( "{{USER ID}}" );
```

Example:

```go
s.SetUserId("alexd");
```

### Set screen resolution with `SetScreenResolution`

If your code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```go
s.SetScreenResolution( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```go
s.SetScreenResolution(1366, 768);
```

### Set viewport dimensions with `SetViewport`

If your code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```go
s.SetViewport( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```go
s.SetViewport(300, 200);
```

### Set color depth with `SetColorDepth`

If your code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```go
s.SetColorDepth( {{BITS PER PIXEL}} );
```

The number should be a positive integer, in bits per pixel. Example:

```go
s.SetColorDepth(32);
```

### Set timezone with `SetTimezone`

This method lets you pass a user's timezone in to Snowplow:

```go
s.timezone( {{TIMEZONE}} );
```

The timezone should be a string:

```go
s.SetColorDepth("Europe/London");
```

### Set the language with `SetLang`

This method lets you pass a user's language in to Snowplow:

```go
s.SetLang( {{LANGUAGE}} );
```

The language should be a string:

```go
s.SetLang('en');
```

### Set the IP Address with `SetIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```go
s.SetIpAddress( {{IP ADDRESS}} );
```

The IP Address should be a string:

```go
s.SetIpAddress('127.0.0.1');
```

### Set the useragent with `SetUseragent`

This method lets you pass a user's useragent string in to Snowplow:

```go
s.SetUseragent( {{USERAGENT}} );
```

The useragent should be a string:

```go
s.SetUseragent('some useragent');
```

### Set the Domain User ID with `SetDomainUserId`

This method lets you pass a user's Domain User ID string in to Snowplow:

```go
s.SetDomainUserId( {{DOMAIN USER ID}} );
```

The Domain User ID should be a string:

```go
s.SetDomainUserId('domain-uid-12');
```

### Set the Domain User ID with `SetNetworkUserId`

This method lets you pass a user's Network User ID string in to Snowplow:

```go
s.SetNetworkUserId( {{NETWORK USER ID}} );
```

The Network User ID should be a string:

```go
s.SetNetworkUserId('network-uid-12');
```

---

# Configure emitters with the Golang tracker
> Configure emitters with collector URI, storage implementation (memory or SQLite3), request types, and byte limits. Set up custom HTTP clients and callbacks for tracking event delivery.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/emitters/

Tracker instances must be initialized with an emitter. This section will go into more depth about the Emitter and how it works under the hood.

The simplest Emitter setup requires only the collector URI to be passed to it:

```go
emitter := sp.InitEmitter(RequireCollectorUri("com.acme"), sp.RequireStorage(*storagememory.Init()))
```

There are other optional builder functions:

| **Function Name**     | **Description**                                | **Required?** | **Default** |
| --------------------- | ---------------------------------------------- | ------------- | ----------- |
| `RequireCollectorUri` | The URI to send events to                      | Yes           | `nil`       |
| `RequireStorage`      | The storage integration to use                 | Yes           | `nil`       |
| `OptionRequestType`   | The request type to use (GET or POST)          | No            | `POST`      |
| `OptionProtocol`      | The protocol to use (http or https)            | No            | `http`      |
| `OptionSendLimit`     | The maximum amount of events to send at a time | No            | `500`       |
| `OptionByteLimitGet`  | The byte limit when sending a GET request      | No            | `40000`     |
| `OptionByteLimitPost` | The byte limit when sending a POST request     | No            | `40000`     |
| `OptionCallback`      | Defines a custom callback function             | No            | `nil`       |
| `OptionHttpClient`    | A custom HTTP client                           | No            | `&Client{}` |

A more complete example:

```go
emitter := sp.InitEmitter(

  sp.RequireCollectorUri("com.acme"),

  sp.RequireStorage(*storagememory.Init()),

  sp.OptionRequestType("GET"),

  sp.OptionProtocol("https"),

  sp.OptionSendLimit(50),

  sp.OptionByteLimitGet(52000),

  sp.OptionByteLimitPost(52000),

  sp.OptionCallback(func(g []CallbackResult, b []CallbackResult) {

    log.Println("Successes: " + IntToString(len(g)))

    log.Println("Failures: " + IntToString(len(b)))

  }),

)
```

As of Version 3 of the Tracker you must pass RequireStorage to set the Storage option for the Emitter. Out of the box we support the following:

- **StorageMemory**: A fully in-memory implementation utilising `go-memdb`
- **StorageSQLite3**: A local disk implementation leveraging `go-sqlite3`

You can also define your own Storage system by implementing the `Storage` interface found in `pkg/storage/storageiface`:

```go
type Storage interface {

  AddEventRow(payload Payload) bool

  DeleteAllEventRows() int64

  DeleteEventRows(ids []int) int64

  GetAllEventRows() []EventRow

  GetEventRowsWithinRange(eventRange int) []EventRow

}
```

To use `StorageMemory` the constructor looks like so:

```go
import storagememory "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/memory"



emitter := sp.InitEmitter(

  sp.RequireCollectorUri("com.acme"),

  sp.RequireStorage(*storagememory.Init()),

  sp.OptionRequestType("GET"),

  sp.OptionProtocol("https"),

  sp.OptionSendLimit(50),

  sp.OptionByteLimitGet(52000),

  sp.OptionByteLimitPost(52000),

  sp.OptionCallback(func(g []CallbackResult, b []CallbackResult) {

    log.Println("Successes: " + IntToString(len(g)))

    log.Println("Failures: " + IntToString(len(b)))

  }),

)
```

To leverage `StorageSQLite3`:

```go
import storagesqlite3 "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/sqlite3"



emitter := sp.InitEmitter(

  sp.RequireCollectorUri("com.acme"),

  sp.RequireStorage(*storagesqlite3.Init("dbname_path.db")),

  sp.OptionRequestType("GET"),

  sp.OptionProtocol("https"),

  sp.OptionSendLimit(50),

  sp.OptionByteLimitGet(52000),

  sp.OptionByteLimitPost(52000),

  sp.OptionCallback(func(g []CallbackResult, b []CallbackResult) {

    log.Println("Successes: " + IntToString(len(g)))

    log.Println("Failures: " + IntToString(len(b)))

  }),

)
```

### Under the hood

Once the emitter receives an event from the Tracker a few things start to happen:

- The event is added to the storage implementation (either `memory` or `sqlite3`)
- A long running go routine is started which will continue to send events as long as they can be found in the database (asynchronous)
- The emitter loop will grab a range of events from the database up until the `SendLimit` as noted \__above_
- The emitter will send all of these events as determined by the Request, Protocol and ByteLimits
  - Each request is sent in its own go routine.
- Once sent it will process the results of all the requests sent and will remove all successfully sent events from the database

**IF** all of the requests failed this loop is terminated eagerly; this is seen as a network failure so attempting to send is a waste of resources. **IF** there are no more events in the database the loop is terminated.

### Builder methods

#### `OptionHttpClient`

An HTTP client can be set with custom settings appropriate for the use-case, such as timeouts and other connection settings.

This method accepts a reference to [http.Client](https://golang.org/pkg/net/http/#Client).

---

# Golang tracker SDK
> The Snowplow Golang Tracker enables server-side event tracking from Go applications. Create subjects, emitters, and trackers with persistent storage to send behavioral events to your Snowplow collector.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/

The [Snowplow Golang Tracker](https://github.com/snowplow/snowplow-golang-tracker) allows you to track Snowplow events from your Golang apps and servers.

There are four basic types of object you will create when using the Snowplow Golang Tracker: subjects, emitters, storage and trackers.

- A subject represents a user whose events are tracked.
- A tracker constructs events and sends them to an emitter.
- The emitter then sends the event to the endpoint you configure, a valid Snowplow Collector, which leverages a storage implementation to store them securely before sending to allow for recovery from failure.

> **Note:** As sending and processing of events is done asynchronously it is advised to create the Tracker as a singleton object. This is due to the fact that all events are first persistently stored in a local Sqlite3 database; if multiple Trackers are created there is the possibility of duplicate sending of events and overt consumption of resources.

---

# Initialize the Golang tracker
> Create a Golang tracker with InitTracker using collector URI and storage implementation. Configure namespace, app ID, platform, and base64 encoding with optional builder functions.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/initialization/

Import the Golang Tracker library like so:

```go
import "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

You will need to refer to the package as `tracker`. If you wish to use something shorter (or if `tracker` is already taken):

```go
import sp "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

The package can now be referred to as `sp` rather than `tracker`.

You will also need to import a `storage` implementation for the `emitter` - you can select from two that we have created or implement your own following the supplied interface.

The `in-memory` or `SQLite3` implementations can be pulled from the following packages:

```go
import storagememory "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/memory" // Maps to tracker.InitStorageMemory() in v2



OR



import storagesqlite3 "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/sqlite3" // Maps to tracker.InitStorageSQLite3(<dbname>) in v2
```

The `Storage` interface can be found in `github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/storageiface`.

That's it - you are now ready to initialize a tracker instance.

## Creating a tracker

The simplest tracker initialization only requires you to provide the URI of the collector to which the tracker will log events and a storage implementation:

```go
import storagememory "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/memory"

import sp "github.com/snowplow/snowplow-golang-tracker/v3/tracker"



emitter := sp.InitEmitter(sp.RequireCollectorUri("com.acme"), sp.RequireStorage(*storagememory.Init()))

tracker := sp.InitTracker(sp.RequireEmitter(emitter))
```

There are other optional builder functions:

| **Function Name**    | **Description**                                                            | **Required?** | **Default** |
| -------------------- | -------------------------------------------------------------------------- | ------------- | ----------- |
| `RequireEmitter`     | The emitter to which events are sent                                       | Yes           | `nil`       |
| `OptionSubject`      | The user being tracked                                                     | No            | `nil`       |
| `OptionNamespace`    | The name of the tracker instance                                           | No            | \`\`        |
| `OptionAppId`        | The application ID                                                         | No            | \`\`        |
| `OptionPlatform`     | The platform the Tracker is running on                                     | No            | `srv`       |
| `OptionBase64Encode` | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64) | No            | `true`      |

A more complete example:

```go
subject := sp.InitSubject()

emitter := sp.InitEmitter(

  sp.RequireCollectorUri("com.acme"),

  sp.RequireStorage(*storagememory.Init()),

)

tracker := sp.InitTracker(

  sp.RequireEmitter(emitter),

  sp.OptionSubject(subject),

  sp.OptionNamespace("namespace"),

  sp.OptionAppId("app-id"),

  sp.OptionPlatform("mob"),

  sp.OptionBase64Encode(false),

)
```

### `RequireEmitter`

Accepts an argument of an Emitter instance pointer; if the object is `nil` will `panic`. See Emitters for more on emitter configuration.

### `OptionSubject`

The user which the Tracker will track. Accepts an argument of a Subject instance pointer.

You don't need to set this during Tracker construction; you can use the `tracker.SetSubject()` method afterwards. In fact, you don't need to create a subject at all. If you don't, though, your events won't contain user-specific data such as timezone and language.

### `OptionNamespace`

If provided, the `namespace` argument will be attached to every event fired by the new tracker. This allows you to later identify which tracker fired which event if you have multiple trackers running.

### `OptionAppId`

The `appId` argument lets you set the application ID to any string.

### `OptionPlatform`

By default we assume the Tracker will be running in a server environment. To override this provide your own platform string.

### `OptionBase64Encode`

By default, unstructured events and custom contexts are encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the Boolean `OptionBase64Encode` function with either `true` or `false` passed in.

---

# Install the Golang tracker
> Install the Golang tracker using go modules with support for Go 1.17.x through 1.19.x. Import the tracker package and start tracking server-side events from your Go applications.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/setup/

The latest version of the Snowplow Golang Tracker is v3.1.0 which has been built and tested using Golang versions `1.19.x`, `1.18.x` and `1.17.x`.

## Setup

All of the tracker code is hosted on [GitHub](https://github.com/snowplow/snowplow-golang-tracker) and versions can be fetched using Golang modules with both `v2` and `v3` available via this method.

To get the package, execute:

```bash
$host go get "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

To import the package, add the following line to your code:

```go
import "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

---

# Track specific events with the Golang tracker
> Track behavioral events including SelfDescribing, ScreenView, PageView, EcommerceTransaction, Structured, and Timing events. Add custom contexts and optional timestamps to all tracked events.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/tracking-specific-events/

Tracking methods supported by the Golang Tracker at a glance:

| **Function**                                                                                           | **Description**                                        |
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| [`TrackSelfDescribingEvent()`](#track-selfdescribingunstructured-events-with-trackselfdescribingevent) | Track a Snowplow custom unstructured event             |
| [`TrackScreenView()`](#track-screen-views-with-trackscreenview)                                        | Track the user viewing a screen within the application |
| [`TrackPageView()`](#track-pageviews-with-trackpageview)                                               | Track and record views of web pages.                   |
| [`TrackEcommerceTransaction()`](#track-ecommerce-transactions-with-trackecommercetransaction)          | Track an ecommerce transaction                         |
| [`TrackStructEvent()`](#track-structured-events-with-trackstructevent)                                 | Track a Snowplow custom structured event               |
| [`TrackTiming()`](#track-timing-events-with-tracktiming)                                               | Track a timing event                                   |

> **Note:** All event structs require pointer values as a way of asserting properly whether or not a value has been passed that might have been required. As such there are three functions provided in `helper` package that allow you to inline pointer values:
>
> - `NewString`
> - `NewInt64`
> - `NewFloat64`
>
> These all accept their respective raw value and return a pointer to this value.
>
> To import the helper package:
>
> ```text
> import sphelp "github.com/snowplow/snowplow-golang-tracker/v3/pkg/common"
> ```

### Common

All events are tracked with specific methods on the tracker instance, of the form `TrackXXX()`, where `XXX` is the name of the event to track.

### Custom contexts

In short, custom contexts let you add additional information about the circumstances surrounding an event in the form of an array of SelfDescribingJson structs (`[]SelfDescribingJson`). Each tracking method accepts an additional optional contexts parameter, which should be a list of contexts:

```go
contextArray := []sp.SelfDescribingJson{}
```

If a visitor arrives on a page advertising a movie, the dictionary for a single custom context in the list might look like this:

```go
contextArray := []sp.SelfDescribingJson{

  *sp.InitSelfDescribingJson(

    "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

    map[string]interface{}{

      "movie_name": "Solaris",

      "poster_country": "JP",

    },

  ),

}
```

This is how to fire a page view event with the above custom context:

```go
tracker.TrackPageView(

  sp.PageViewEvent{

    PageUrl: sphelp.NewString("acme.com"),

    Contexts: contextArray,

  },

)
```

Note that even though there is only one custom context attached to the event, it still needs to be placed in an array.

### Optional timestamp argument

Each `Track...()` method supports an optional timestamp argument; this allows you to manually override the timestamp attached to this event. The timestamp should be in milliseconds since the Unix epoch.

If you do not pass this timestamp in as an argument, then the Golang Tracker will use the current time to be the timestamp for the event.

Here is an example tracking a structured event and supplying the optional timestamp argument:

```go
tracker.TrackStructEvent(sp.StructuredEvent{

  Category: sphelp.NewString("some category"),

  Action: sphelp.NewString("some action"),

  Timestamp: sphelp.NewInt64(1368725287000),

})
```

### Optional true-timestamp argument

Each `Track...()` method supports an optional true-timestamp argument; this allows you to provide the true-timestamp attached to this event to help with the timing of events in multiple timezones. The timestamp should be in milliseconds since the Unix epoch.

Here is an example tracking a structured event and supplying the optional true-timestamp argument:

```go
tracker.TrackStructEvent(sp.StructuredEvent{

  Category: sphelp.NewString("some category"),

  Action: sphelp.NewString("some action"),

  TrueTimestamp: sphelp.NewInt64(1368725287000),

})
```

### Optional event ID argument

Each `Track...()` method supports an optional event id argument; this allows you to manually override the event ID attached to this event. The event ID should be a valid version 4 UUID string.

Here is an example tracking a structured event and supplying the optional event ID argument:

```go
tracker.TrackStructEvent(sp.StructuredEvent{

  Category: sphelp.NewString("some category"),

  Action: sphelp.NewString("some action"),

  EventId: sphelp.NewString("486820fb-e722-4311-b33d-d2f319b511f6"),

})
```

### Optional Subject argument

Each `Track...()` method supports an optional Subject argument; this allows you to manually override the Subject information that is usually pulled from the `Tracker` object.

Here is an example of tracking a Page View event and supplying the optional Subject argument:

```go
eventSubject := sp.InitSubject()

eventSubject.SetUserId("987654321")



tracker.TrackPageView(sp.PageViewEvent{

	PageUrl:  sphelp.NewString("acme.com"),

	Subject:  eventSubject,

})
```

### Track SelfDescribing/Unstructured events with `TrackSelfDescribingEvent()`

Use `TrackSelfDescribingEvent()` to track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

The arguments are as follows:

| **Argument**    | **Description**               | **Required?** | **Validation**        |
| --------------- | ----------------------------- | ------------- | --------------------- |
| `Event`         | The properties of the event   | Yes           | \*SelfDescribingJson  |
| `Timestamp`     | When the event occurred       | No            | \*int64               |
| `EventId`       | The event ID                  | No            | \*string              |
| `TrueTimestamp` | The true time of event        | No            | \*int64               |
| `Contexts`      | Custom contexts for the event | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject        | No            | Subject               |

Example:

```go
// Create a data map

data := map[string]interface{}{

  "level": 5,

  "saveId": "ju302",

  "hardMode": true,

}



// Create a new SelfDescribingJson

sdj := sp.InitSelfDescribingJson("iglu:com.example_company/save-game/jsonschema/1-0-2", data)



tracker.TrackSelfDescribingEvent(sp.SelfDescribingEvent{

  Event: sdj,

})
```

For more on JSON schema, see the [blog post](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

### Track screen views with `TrackScreenView()`

Use `TrackScreenView()` to track a user viewing a screen (or equivalent) within your app:

| **Argument**    | **Description**                     | **Required?** | **Type**              |
| --------------- | ----------------------------------- | ------------- | --------------------- |
| `Name`          | Human-readable name for this screen | No            | \*string              |
| `Id`            | Unique identifier for this screen   | No            | \*string              |
| `Timestamp`     | When the event occurred             | No            | \*int64               |
| `EventId`       | The event ID                        | No            | \*string              |
| `TrueTimestamp` | The true time of event              | No            | \*int64               |
| `Contexts`      | Custom contexts for the event       | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject              | No            | Subject               |

Although name and id are not individually required, at least one must be provided or the event will fail validation.

Example:

```go
tracker.TrackScreenView(sp.ScreenViewEvent{

  Id: sphelp.NewString("Screen ID"),

})
```

### Track pageviews with `TrackPageView()`

Use `TrackPageView()` to track a user viewing a page within your app:

| **Argument**    | **Description**                      | **Required?** | **Validation**        |
| --------------- | ------------------------------------ | ------------- | --------------------- |
| `PageUrl`       | The URL of the page                  | Yes           | \*string              |
| `PageTitle`     | The title of the page                | No            | \*string              |
| `Referrer`      | The address which linked to the page | No            | \*string              |
| `Timestamp`     | When the event occurred              | No            | \*int64               |
| `EventId`       | The event ID                         | No            | \*string              |
| `TrueTimestamp` | The true time of event               | No            | \*int64               |
| `Contexts`      | Custom contexts for the event        | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject               | No            | Subject               |

Example:

```go
tracker.TrackPageView(sp.PageViewEvent{

  PageUrl: sphelp.NewString("acme.com"),

})
```

### Track ecommerce transactions with `TrackEcommerceTransaction()`

Use `TrackEcommerceTransaction()` to track an ecommerce transaction:

| **Argument**    | **Description**                 | **Required?** | **Validation**                   |
| --------------- | ------------------------------- | ------------- | -------------------------------- |
| `OrderId`       | ID of the eCommerce transaction | Yes           | \*string                         |
| `TotalValue`    | Total transaction value         | Yes           | \*float64                        |
| `Affiliation`   | Transaction affiliation         | No            | \*string                         |
| `TaxValue`      | Transaction tax value           | No            | \*float64                        |
| `Shipping`      | Delivery cost charged           | No            | \*float64                        |
| `City`          | Delivery address city           | No            | \*string                         |
| `State`         | Delivery address state          | No            | \*string                         |
| `Country`       | Delivery address country        | No            | \*string                         |
| `Currency`      | Transaction currency            | No            | \*string                         |
| `Items`         | Items in the transaction        | Yes           | \[]EcommerceTransactionItemEvent |
| `Timestamp`     | When the event occurred         | No            | \*int64                          |
| `EventId`       | The event ID                    | No            | \*string                         |
| `TrueTimestamp` | The true time of event          | No            | \*int64                          |
| `Contexts`      | Custom contexts for the event   | No            | \[]SelfDescribingJson            |
| `Subject`       | Event specific Subject          | No            | Subject                          |

The `items` argument is an Array of TransactionItems. `TrackEcommerceTransaction` fires multiple events: one transaction event for the transaction as a whole, and one transaction item event for each element of the `Items` list. Each transaction item event will have the same timestamp, true timestamp, order ID, and currency as the main transaction event.

These are the fields with which a TransactionItem can be created.

| **Field**  | **Description**               | **Required?** | **Validation**        |
| ---------- | ----------------------------- | ------------- | --------------------- |
| `Sku`      | Item SKU                      | Yes           | \*string              |
| `Price`    | Item price                    | Yes           | \*float64             |
| `Quantity` | Item quantity                 | Yes           | \*int64               |
| `Name`     | Item name                     | No            | \*string              |
| `Category` | Item category                 | No            | \*string              |
| `EventId`  | The event ID                  | No            | \*string              |
| `Contexts` | Custom contexts for the event | No            | \[]SelfDescribingJson |
| `Subject`  | Event specific Subject        | No            | Subject               |

Example of tracking a transaction containing two items:

```go
items := []sp.EcommerceTransactionItemEvent{

  sp.EcommerceTransactionItemEvent{

    Sku: sphelp.NewString("pbz0026"),

    Price: sphelp.NewFloat64(20),

    Quantity: sphelp.NewInt64(1),

  },

  sp.EcommerceTransactionItemEvent{

    Sku: sphelp.NewString("pbz0038"),

    Price: sphelp.NewFloat64(15),

    Quantity: sphelp.NewInt64(1),

    Name: sphelp.NewString("red hat"),

    Category: sphelp.NewString("menswear"),

  },

}



tracker.TrackEcommerceTransaction(sp.EcommerceTransactionEvent{

  OrderId: sphelp.NewString("6a8078be"),

  TotalValue: sphelp.NewFloat64(35),

  Affiliation: sphelp.NewString("some-affiliation"),

  TaxValue: sphelp.NewFloat64(6.12),

  Shipping: sphelp.NewFloat64(30),

  City: sphelp.NewString("Dijon"),

  State: sphelp.NewString("Bourgogne"),

  Country: sphelp.NewString("France"),

  Currency: sphelp.NewString("EUR"),

  Items: items,

})
```

### Track structured events with `TrackStructEvent()`

Use `TrackStructEvent()` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument**    | **Description**                                                  | **Required?** | **Validation**        |
| --------------- | ---------------------------------------------------------------- | ------------- | --------------------- |
| `Category`      | The grouping of structured events which this `action` belongs to | Yes           | \*string              |
| `Action`        | Defines the type of user interaction which this event involves   | Yes           | \*string              |
| `Label`         | A string to provide additional dimensions to the event data      | No            | \*string              |
| `Property`      | A string describing the object or the action performed on it     | No            | \*string              |
| `Value`         | A value to provide numerical data about the event                | No            | \*float64             |
| `Timestamp`     | When the event occurred                                          | No            | \*int64               |
| `EventId`       | The event ID                                                     | No            | \*string              |
| `TrueTimestamp` | The true time of event                                           | No            | \*int64               |
| `Contexts`      | Custom contexts for the event                                    | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject                                           | No            | Subject               |

Example:

```go
tracker.TrackStructEvent(sp.StructuredEvent{

  Category: sphelp.NewString("shop"),

  Action: sphelp.NewString("add-to-basket"),

  Property: sphelp.NewString("pcs"),

  Value: sphelp.NewFloat64(2),

})
```

### Track timing events with `TrackTiming()`

Use `TrackTiming()` to track a timing event.

The arguments are as follows:

| **Argument**    | **Description**               | **Required?** | **Validation**        |
| --------------- | ----------------------------- | ------------- | --------------------- |
| `Category`      | The category of the event     | Yes           | \*string              |
| `Variable`      | The variable of the event     | Yes           | \*string              |
| `Timing`        | The timing of the event       | Yes           | \*int64               |
| `Label`         | The label of the event        | No            | \*string              |
| `Timestamp`     | When the event occurred       | No            | \*int64               |
| `EventId`       | The event ID                  | No            | \*string              |
| `TrueTimestamp` | The true time of event        | No            | \*int64               |
| `Contexts`      | Custom contexts for the event | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject        | No            | Subject               |

Example:

```go
tracker.TrackTiming(sp.TimingEvent{

  Category: sphelp.NewString("Timing Category"),

  Variable: sphelp.NewString("Some var"),

  Timing: sphelp.NewInt64(124578),

})
```

---

# Google AMP tracker
> Integrate Snowplow tracking into Google AMP (Accelerated Mobile Pages) using amp-analytics. Track page views, structured events, page pings, and custom events from mobile-optimized AMP pages with cross-domain session linking.
> Source: https://docs.snowplow.io/docs/sources/google-amp-tracker/

Snowplow is collaborating with Google on their [Accelerated Mobile Pages Project](https://www.ampproject.org/) (AMPP or AMP for short). AMP is an open source (Apache License 2.0) initiative to improve the mobile web experience by optimizing web pages for mobile devices.

To learn more about analytics for AMP pages see the [amp-analytics](https://www.ampproject.org/docs/reference/extended/amp-analytics.html) reference. For general information about AMP see [What Is AMP?](https://www.ampproject.org/docs/get_started/about-amp.html) on the [Accelerated Mobile Pages (AMP) Project](https://www.ampproject.org/) site.

Snowplow is natively integrated into the project, so pages optimized with AMP HTML can be tracked in Snowplow by adding the appropriate `amp-analytics` tag to your pages:

```html


<body>

  ...

  <amp-analytics type="snowplow_v2" id="snowplow_v2">

  <script type="application/json">

  {

    "vars": {

     "collectorHost": "collector.snplow.net",

     "appId": "amp-examples-v2.0.2",

     ...

    },

    "linkers": {

     "enabled": true,

     ...

    },

    "triggers": {

     "defaultPageview": {

       "on": "visible",

       "request": "pageView"

     }

    }

  }

  </script>

  </amp-analytics>

</body>
```

> **Note:** The tracker utilises AMP linker functionality to ensure that user identification can be done where a user may visit via the google domain or the site's own domain. In order to function, this requires that linkers are enabled in the amp-analytics configuration. Not doing so can result in changing AMP client Ids (which are the primary user identifier).

## Standard variables

`collectorHost` and `appId` must be provided in the `"vars"` section of the tag:

```javascript
"vars": {

  ...

},
```

The rest are optional

### `collectorHost`

Specify the host to your collector like so:

```javascript
"vars": {

  "collectorHost": "snowplow-collector.acme.com",

  ...
```

Notes:

- Do _not_ include the protocol aka schema (`http(s)://`)
- Do _not_ include a trailing slash
- Use of HTTPS is mandatory in AMP, so your Snowplow collector **must** support HTTPS

### `appId`

You must set the application ID for the website you are tracking via AMP:

```javascript
"vars": {

  "appId": "campaign-microsite",

  ...
```

Notes:

- You do not have to use the `appId` to distinguish AMP traffic from other web traffic (unless you want to) - see the [Analytics](#analytics) section for an alternative approach.

### `userId`

Specify the optional `"userId"` var to set the uid/user\_id [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#user-fields) field.

```javascript
"vars": {

  "userId": "someUserId",

  ...
```

### `nameTracker`

Specify the optional "nameTracker" var to set the tna/name\_tracker [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields) field.

```javascript
"vars": {

  "nameTracker": "someTrackerName",

  ...
```

### `customContexts`

Custom contexts may be added by including Self-Describing JSON as a `"customContexts"` var, with `"` characters escaped:

```javascript
"vars": {

  "customContexts":  "{\"schema\":\"iglu:com.acme/first_context/jsonschema/1-0-0\",\"data\":{\"someKey\":\"someValue\"}}"

  ...
```

Multiple custom contexts may be added by separating each self-describing JSON with a comma:

```javascript
"vars": {

  "customContexts":  "{\"schema\":\"iglu:com.acme/first_context/jsonschema/1-0-0\",\"data\":{\"someKey\":\"someValue\"}},{\"schema\":\"iglu:com.acme/second_context/jsonschema/1-0-0\",\"data\":{\"someOtherKey\":\"someOtherValue\"}}"

  ...
```

Custom contexts may either be set globally for all events (as a top-level var, see above) or per-event trigger:

```javascript
...

"triggers": {

...

  "defaultPageview": {

    "on": "click",

    "selector": "visible",

    "request": "pageView",

    "vars": {

      "customContexts":  "{\"schema\":\"iglu:com.acme/first_context/jsonschema/1-0-0\",\"data\":{\"someKey\":\"someValue\"}}"

    }

  }

}

...
```

These approaches may not currently be mixed, however.

## Tracking events

The following trigger request values are supported for the Snowplow Analytics configuration:

- `pageView` for page view tracking
- `structEvent` for structured event tracking
- `ampPagePing` for page ping tracking
- `selfDescribingEvent` for custom event tracking

All event tracking is disabled by default; you can enable it on an event-by-event basis as follows:

### Page view

Enable the page view tracking like so:

```javascript
<amp-analytics type="snowplow_v2" id="event1">

<script type="application/json">

{

  "vars": {

    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host

    "appId": "campaign-microsite"                    // Replace with your app ID

  },

  "triggers": {

    "trackPageview": {  // Trigger names can be any string. trackPageview is not a required name

      "on": "visible",

      "request": "pageView"

    }

  }

}

</script>

</amp-analytics>
```

### Structured events

Structured events are user interactions with content that can be tracked independently from a web page or a screen load. "Structured" refers to the Google Analytics-style structure of having up to five fields (with only the first two required).

Events can be sent by setting the AMP trigger request value to event and setting the required event category and action fields.

The following example uses the selector attribute of the trigger to send an event when a particular element is clicked:

```javascript
<amp-analytics type="snowplow_v2" id="event2">

<script type="application/json">

{

  "vars": {

    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host

    "appId": "campaign-microsite"                    // Replace with your app ID

  },

  "triggers": {

    "trackClickOnHeader" : {

      "on": "click",

      "selector": "#header",

      "request": "structEvent",

      "vars": {

        "structEventCategory": "ui-components",

        "structEventAction": "header-click"

      }

    }

  }

}

</script>

</amp-analytics>
```

You can set key-value pairs for the following event fields in the vars attribute of the trigger:

| **Argument**          | **Description**                                                  | **Required?** | **Validation** |
| --------------------- | ---------------------------------------------------------------- | ------------- | -------------- |
| `structEventCategory` | The grouping of structured events which this `action` belongs to | Yes           | String         |
| `structEventAction`   | Defines the type of user interaction which this event involves   | Yes           | String         |
| `structEventLabel`    | A string to provide additional dimensions to the event data      | No            | String         |
| `structEventProperty` | A string describing the object or the action performed on it     | No            | String         |
| `structEventValue`    | A value to provide numerical data about the event                | No            | Int or Float   |

### Page pings

Enable page ping tracking like so:

```javascript
<amp-analytics type="snowplow_v2" id="event3">

<script type="application/json">

{

  "vars": {

    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host

    "appId": "campaign-microsite"                    // Replace with your app ID

  },

  "triggers": {

    "trackPagePings": {

      "on": "timer",

      "request": "ampPagePing",

      "timerSpec": {

        "interval": 30,      // Set to desired ping interval

        "maxTimerLength": 1800,

        "immediate": false,

        "startSpec": {

          "on": "visible",

          "selector": ":root"

        },

        "stopSpec": {

          "on": "hidden",

          "selector": ":root"

        }

      }

    }

  }

}

</script>

</amp-analytics>
```

AMP page ping events will be sent as AMP-specific page ping events (rather than Javascript tracker page pings), against the [AMP page ping schema](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_page_ping/jsonschema/1-0-0), since the data available on AMP is defined differently to Javascript tracker page pings.

All events are sent with an [AMP web page](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_web_page/jsonschema/1-0-0) context, for aggregation of pings by page view id.

### Custom events

Custom events may be sent via the AMP tracker by passing the schema vendor, and version, and an escaped JSON of the desired data, as follows:

```javascript
<amp-analytics type="snowplow_v2" id="event3">

<script type="application/json">

{

  "vars": {

    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host

    "appId": "campaign-microsite"                    // Replace with your app ID

  },

  "triggers": {

    "trackSelfDescribingEvent": {

      "on": "click",

      "selector": "a",

      "request": "selfDescribingEvent",

      "vars": {

        "customEventSchemaVendor": "com.snowplowanalytics.snowplow",

        "customEventSchemaName": "link_click",

        "customEventSchemaVersion": "1-0-1",

        "customEventSchemaData": "{\"targetUrl\":\"${targetUrl}\",\"elementId\":\"TEST\",\"elementContent\":\"${elementContent}\"}"

      }

    }

  }

}

</script>

</amp-analytics>
```

## Analytics

v2 of the tracker brings with it significant improvements in our ability to model and gain insights.

### Standard fields

All events sent via this tracker will have:

- `v_tracker` set to `amp-1.1.0`
- `platform` set to `web`

If you want to analyze events sent via this tracker, you may prefer to query for `v_tracker LIKE 'amp-%'` to future-proof your query against future releases of this tracker (which may change the version number).

### Page view and ping aggregation

By default, the [AMP web page context](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_web_page/jsonschema/1-0-0) is attached to every event. This will contain the AMP-defined `PAGE_VIEW_ID_64`, which defined as "intended to be random with a high entropy and likely to be unique per URL, user and day".

Users can aggregate page views, page pings and other events on-page by this ID to aggregate engaged time, and model events to a page view level, by combining it with the url, amp client ID, and date.

Note that page pings and the page view ID itself are not defined by Snowplow's logic, but by what's made available by AMP - therefore applying the same logic to this data as that produced by the Javascript tracker is liable to produce different results.

### Session information

> **Note:** Available from version 1.1.0.

By default the [AMP Session](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_session/jsonschema/) context is attached to every event. This context allows for tracking information related to session analytics capabilities, as implemented in the AMP framework. The attributes included are the following:

- `ampSessionId`: An identifier for the AMP session.
- `ampSessionIndex`: The index of the current session for this user.
- `sessionEngaged`: If there has been any kind of user engagement in the AMP session. Engagement in this context means if the page is visible, has focus and is in the foreground.
- `sessionCreationTimestamp`: Timestamp at which the session was created in milliseconds elapsed since the UNIX epoch.
- `lastSessionEventTimestamp`: Timestamp at which the last event took place in the session in milliseconds elapsed since the UNIX epoch.

### User Identification

By default, the [AMP ID](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_id/jsonschema/1-0-0) context is attached to every event. This contains the AMP Client ID, the `user_id` (if set via the `userId` var), and the domain\_userid (if passed to an AMP page via cross-domain linking - more detail below).

This provides a map between the main relevant identifiers, which can be used to model user journeys across platforms. Users can choose to instrument further user identification methods using custom contexts.

In order for the AMP Client ID to behave as expected, the 'linker' parameter of the configuration JSON must have `'enabled': true` set. Failing to do so can result in changing AMP client IDs, even where a user remains on AMP pages for the whole journey. (This can happen because AMP pages can be served from Google's AMP cache).

The tracker is designed to handle user journeys as follows:

#### JS-tracker page to AMP page

Where a user moves from a standard web page, tracked by the Javascript tracker, to an AMP page, the domain userid from the Javascript tracker can be passed to the AMP tracker by enabling the [Javascript tracker's crossDomainLinker](/docs/sources/web-trackers/tracker-setup/initialization-options/). The AMP tracker will parse the value from the querystring, and attach it to all events, along with the AMP client ID, via the AMP ID context.

The AMP tracker uses a combination of cookies and the AMP linker to attempt to retain the value, however due to the nature of AMP pages, there is no guarantee that the value will be retained across sessions. To ensure best possible retention of the value within the session, make sure the tracker config has linker pages enabled for your AMP domains:

```javascript
...

"linkers": {

  "enabled": true,

  "proxyOnly": false,

  "destinationDomains": ["ampdomain"]

},

...
```

Data models should be designed for an 'at least once' identification structure - in other words, configuring the cross-domain linker correcttly will ensure at least one event contains both AMP ID and domain User ID. Models should aim to map this across all events. Of course, this method of identification depends on the user traveling directly from a JS-tracked page to an AMP page at least once.

#### AMP page to JS-tracker page

Where a user moves frrom an AMP page to a standard web page which is tracked by the Javascript tracker, the AMP tracker will use AMP's linker functionality to append the AMP CLient ID to the querystring, as long as linkers are enabled for the destination domain:

```javascript
...

"linkers": {

  "enabled": true,

  "proxyOnly": false,

  "destinationDomains": ["destDomain"]

},

...
```

This will add a querystring parameter 'linker=' to the destination url, which contains the amp\_id value, base-64 encoded. This will look something like this: `?sp_amp_linker=1*1c1wx43*amp_id*amp-a1b23cDEfGhIjkl4mnoPqr`.

The structure of this param is explained in the [AMP documentation](https://github.com/ampproject/amphtml/blob/master/extensions/amp-analytics/linker-id-receiving.md) - models can extract the base64-encoded AMP Client ID, decode it, and map it to the domain userid (or any other user value) from the Javascript tracker.

## Reporting Issues and Contributing

A fork of the AMP project can be found in the [Snowplow Incubator Github](https://github.com/snowplow-incubator/amphtml) Repo. Please submit issues, bugreports and PRs to this repo.

---

# Configure the Snowplow Ecommerce Tag
> Configure the Snowplow Ecommerce Tag Template in GTM using native Snowplow Ecommerce API or transitional GA4/UA adapter APIs. Set tracking parameters, custom contexts, and product data for ecommerce event tracking.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/ecommerce-tag-template/configuration/

Use the native Snowplow Ecommerce API or [transitional GA4/UA ecommerce adapter APIs](/docs/sources/web-trackers/tracking-events/ecommerce/#ga4ua-ecommerce-transitional-api) for existing dataLayer implementations using those formats. To get full value from the [Snowplow Ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/) we recommend using the native API when possible.

![Ecommerce API dropdown with three options: Snowplow Ecommerce (selected and highlighted), Google Analytics 4 Ecommerce, and Universal Analytics Enhanced Ecommerce.](/assets/images/01_ecommerce_api-5ccc2f11a8b43b04dbc188a27da29ea1.png)

## Tracking Parameters

**Snowplow Ecommerce:**

![Snowplow Ecommerce Tracking Parameters section showing Ecommerce Function set to Track Product View, an empty Ecommerce Argument field, an empty Custom Context Entities table with an Add Row button, and Set Custom Timestamp set to Use system time.](/assets/images/02_sp_tracking_parameters-79d633f456f8c14286ec3ed73e2fd3fb.png)

#### Snowplow Ecommerce Function

In this section you can select the [Snowplow Ecommerce function](/docs/sources/web-trackers/tracking-events/ecommerce/) to use.

#### Snowplow Ecommerce Argument

In this textbox you can specify the argument to the ecommerce function. This can be a Variable that evaluates to a corresponding object.

#### Additional Tracking Parameters

##### Add Custom Context Entities

Use this table to attach [custom context entities](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-entity) to the Snowplow event. Each row can be set to a Google Tag Manager variable that returns an array of custom contexts to add to the event hit.

##### Set Custom Timestamp

Set this to a UNIX timestamp in case you want to [override the default timestamp](/docs/sources/web-trackers/tracking-events/#custom-timestamp) used by Snowplow.

**GA4 Ecommerce:**

![GA4 Ecommerce Tracking Parameters section showing GA4 Ecommerce Function set to Track GA4 View Item List, and Ecommerce Arguments subsection with empty DataLayer ecommerce and Options object fields.](/assets/images/02_ga4_tracking_parameters-62661b55514ba47b36edf5fbc50c3282.png)

#### GA4 Ecommerce Function

In this section you can select the [Google Analytics 4 Ecommerce function](/docs/sources/web-trackers/tracking-events/ecommerce/) to use.

#### GA4 Ecommerce Arguments

##### DataLayer ecommerce

Here you can specify the dataLayer ecommerce variable to use, i.e. a variable that returns the `ecommerce` object itself.

##### Options object

Here you can specify a variable returning an object holding additional information for the ecommerce event (e.g. including `currency`, `finalCartValue`, `step`, etc).

**Universal Analytics Enhanced Ecommerce:**

![Universal Analytics Enhanced Ecommerce Tracking Parameters section showing UA Enhanced Ecommerce Function set to Track Enhanced Ecommerce Product List View, and Ecommerce Arguments subsection with empty DataLayer ecommerce and Options object fields.](/assets/images/02_ua_tracking_parameters-1c48c247c61794f48eac816ad435b34c.png)

#### Universal Analytics Enhanced Ecommerce Function

In this section you can select the [Universal Analytics Enhanced Ecommerce function](/docs/sources/web-trackers/tracking-events/ecommerce/) to use.

#### Universal Analytics Enhanced Ecommerce Arguments

##### DataLayer ecommerce

Here you can specify the dataLayer ecommerce variable to use.

##### Options object

Here you can specify a variable returning an object holding additional information for the ecommerce event (e.g.including currency, finalCartValue, step etc).

***

## Snowplow Tracker and Ecommerce Plugin Settings

![Snowplow Tracker and Ecommerce Plugin Settings section showing a Tracker Settings dropdown prompting to select a Snowplow v3 Settings variable, and Plugin Settings with Snowplow Ecommerce Plugin Library set to Self-hosted and an empty Self-hosted Plugin URL field.](/assets/images/04_tracker_plugin_settings-383e984e9035913a2f284e466f2723fe.png)

### Tracker Settings

The Snowplow Ecommerce tag template **requires** a Snowplow Settings Variable to be setup. In this section you can select the Google Tag Manager variable of type [Snowplow Settings](/docs/sources/google-tag-manager/quick-start/) to use.

### Plugin Settings

In this section you can select how the plugin will be added. The available options are:

- `jsDelivr`: To get the plugin URL from jsDelivr CDN. Choosing this option allows you to specify the plugin version to be used.
- `unpkg`: To get the plugin URL from unpkg CDN. Choosing this option allows you to specify the plugin version to be used.
- `Self-hosted`: To get the plugin library from a specified URL. This option requires a [Permission](https://developers.google.com/tag-platform/tag-manager/templates/permissions) change to allow injecting the plugin script from the specified URL.
- `Do not add`: To not add the plugin (e.g. when using a [Custom Bundle](/docs/sources/web-trackers/plugins/configuring-tracker-plugins/) with the plugin already included). The default plugins bundled with the JavaScript tracker has changed from v3 to v4. Ensure that all plugins that you require are included. A list of the default plugins is available [here](/docs/sources/web-trackers/plugins/).

---

# Snowplow GTM Ecommerce template
> Install and configure the Snowplow Ecommerce Tag Template in Google Tag Manager to track product views, transactions, cart actions, and checkout events using the Snowplow Ecommerce plugin for v3 and v4 trackers.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/ecommerce-tag-template/

The Ecommerce Template is a separate Tag Template that can be added to your GTM workspace to track ecommerce events. This was done to keep the main Snowplow Tag Template a bit lighter, and for ease of tag management.

This template implements the [Snowplow Ecommerce plugin](/docs/sources/google-tag-manager/ecommerce-tag-template/) for the Snowplow JavaScript tracker and can be used alongside either [v3](/docs/sources/google-tag-manager/previous-versions/) or [v4](/docs/sources/google-tag-manager/) of the Snowplow tag.

## Template Installation

### Tag Manager Template Gallery

Search for "Snowplow Ecommerce v3" in the [Tag Manager Template Gallery](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-ecommerce-v3) and click `Add to Workspace`.

### Manual Installation

1. Download [template.tpl](https://github.com/snowplow/snowplow-gtm-tag-template-ecommerce-v3)
2. Create a new Tag template in the Templates section of your GTM container
3. Click the More Actions menu and select Import
4. Import the `template.tpl` file downloaded in Step 1
5. Click Save

## Tag Setup

With the template installed, you can now add the Snowplow Ecommerce Tag to your GTM Container.

1. From the Tag tab, select `New`, then select the Snowplow Ecommerce Tag as your tag type
2. Select your desired Trigger for the ecommerce events you want to track
3. [Configure the Tag](/docs/sources/google-tag-manager/ecommerce-tag-template/configuration/)
4. Click Save

---

# Snowplow Google Tag Manager templates
> Deploy the Snowplow JavaScript tracker through Google Tag Manager using custom templates for v4. Configure event tracking, ecommerce, and custom variables with server-side and client-side GTM tags.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/

Using the Snowplow GTM custom templates you can deploy, implement, and configure the Snowplow [JavaScript tracker](/docs/sources/web-trackers/) directly on the website using Google Tag Manager.

The main Tag template that you will need to use when setting up the JavaScript Tracker v4 in GTM is available in the [Tag Manager Template Gallery](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-v4). To setup the Snowplow v4 Tag, you will also need the Snowplow v4 Settings Variable template. The templates you will need are:

1. [Snowplow v4](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-v4): Load, configure, and deploy the Snowplow JavaScript tracker library. It supports the full functionality of the JavaScript SDK.
2. [Snowplow v4 Settings](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-variable-template-v4): A variable template which can be used to easily apply a set of tracker configuration parameters to tags created with the Snowplow v4 tag template.

For Ecommerce tracking, the [Snowplow Ecommerce Tag](https://github.com/snowplow/snowplow-gtm-tag-template-ecommerce-v3) is available on GitHub.

---

# Quick start guide for using Snowplow in Google Tag Manager
> Add Snowplow v4 custom templates to your Google Tag Manager workspace and configure basic tracking. Import templates from the GTM gallery and set up your collector endpoint for event tracking.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/quick-start/

This guide will walk you through the initial setup for Snowplow in Google Tag Manager.

## Adding the Templates

To get started with Snowplow in Google Tag Manager, you will need to add the Snowplow Tag Template and the Snowplow Settings Variable Template to your GTM workspace.

Links to both templates can be found [here](/docs/sources/google-tag-manager/).

### Snowplow Tag Template

1. Navigate to the `Templates` tab in your GTM workspace and click the `Search Gallery` button in the `Tag Templates` section.

![GTM Templates tab showing the Tag Templates section with the Search Gallery button highlighted and an empty state, and the Variable Templates section below it also with a Search Gallery button.](/assets/images/tag-template-search-0ec39a0b52c7894a89449492ad0308ce.png)

2. Search for "Snowplow v4"

![Community Template Gallery search panel with \&quot;Snowplow v4\&quot; entered in the search field, showing one result: Snowplow v4 by snowplow with a green verified checkmark.](/assets/images/search-27677eb5bfee0e80f905e1f996b157ce.png)

3. Click on the template, and then click `Add to Workspace` in the next screen. Review the permissions and click `Add` to finalize the import.

### Snowplow Settings Variable Template

The Snowplow Settings Variable template is used to configure the Snowplow tracker, such as the collector endpoint, privacy options, and tracker version. Although possible to use the Snowplow tag without the settings variable, it's highly recommended to use it for ease of configuration, along with keeping the tracker configuration separate from the tag.

1. Again in the `Templates` tab in your GTM workspace, click the `Search Gallery` button in the `Variable Templates` section.

![GTM Templates tab with the Search Gallery button in the Variable Templates section highlighted, and the Tag Templates section above it showing an empty state.](/assets/images/variable-template-search-2d20f633b21d310f8664a688d210f0d3.png)

2. Search for "Snowplow v4 Settings"

3. Click on the template, and then click `Add to Workspace` in the next screen. No permissions are required, so click `Add` to finalize the import.

## Configuring the Settings Variable

1. Navigate to the `Variables` tab in your GTM workspace and click `New` in `User-Defined Variables`.

![GTM Variables tab showing built-in variables (Event, Page Hostname, Page Path, Page URL, and Referrer) and the User-Defined Variables section with the New button highlighted and an empty state.](/assets/images/variables-new-c32b98d0e9bac37687e92d6234177414.png)

2. Select `Snowplow v4 Settings` from the list of available variables.

3. Under `Tracker Options`, enter your Snowplow collector endpoint set up when [configuring your collector](/docs/pipeline/collector/).

![GTM Variable Configuration panel for Untitled Variable showing type set to Snowplow v4 Settings, Tracker Options with Tracker Name spTracker and Collector Endpoint Hostname collector.mydomain.com, and JavaScript Tracker with library set to jsDelivr and version 4.0.2.](/assets/images/variable-9147f9ce6c3ee4fe6ed7770f3eee1fa7.png)

> **Info:** You might consider using conditional variables to set the Collector endpoint based on the environment, e.g. sending data to [Snowplow Micro](/docs/testing/snowplow-micro/) during development.

4. Under `JavaScript Tracker`, choose a hosting option. To get started quickly, select either `unpkg` or `jsDelivr` and enter a library version.

5. Give your variable a name and click `Save`.

## Implementing the Snowplow Tag

In this section, we will create a simple tag to fire a page view event.

1. Navigate to the `Tags` tab in your GTM workspace and click `New`.

![GTM Tags tab with the New button highlighted and an empty state message indicating no tags have been created yet.](/assets/images/new-tag-e29e5a0837ec41a24d07c0bf2c89e5af.png)

2. Click on the `Tag Configuration` section and select `Snowplow v4`.

3. Set the `Tag Type` to `Page View`, if it is not already selected.

4. Under `Tracker Initialisation`, select the Snowplow Settings variable we created earlier.

5. Add a trigger to the tag. This will determine when the tag is fired. For a page view tag, you can use the built-in `All Pages` trigger.

6. Give your tag a name and click `Save`.

## Testing the Tag

To test the tag, you can use the GTM preview mode. Click the `Preview` button in the top right of the GTM interface. This will open a new tab with your website and the GTM preview console.

Ensure that you see the Page View event in your Snowplow pipeline. If you don't have a full pipeline set up yet, you can use [Snowplow Micro](/docs/testing/snowplow-micro/) or [Snowplow Inspector](/docs/testing/snowplow-inspector/) to check that the event is sent correctly.

---

# Snowplow GTM Settings template
> Configure the Snowplow Settings Variable template for GTM with tracker options, collector endpoints, privacy settings, cookies, dispatching methods, and predefined context entities for consistent tracking configuration.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/settings-template/

This page describes the settings available in the Snowplow Settings Variable template for Google Tag Manager.

## Tracker Options

### Tracker Name

It is important to set the tracker name. The reason you might have more than one tracker name generated on the site is if you have different configuration objects or tracking endpoints to which you want to send commands.

When the tag runs, it first checks if a tag with this name has already been initialized. If it has, it then proceeds to send the command to this tracker name. If a tracker with this name has _not_ been initialized, a new tracker is initialized with the tracker configuration from this settings variable.

This means that a tracker configuration is applied **only once** to the tracker. Thus if you have more than one tag running on the site, each with the same tracker name but different tracker configurations, only the configuration of the tag that fires _first_ will be applied to the tracker.

### Collector Endpoint Hostname

This needs to be set to the hostname/domain (e.g. `sp.domain.com`) on which you’ve configured your [Snowplow Collector](/docs/pipeline/collector/)

## JavaScript Tracker

### Snowplow JavaScript Tracker Library

This determines the source of the Snowplow JavaScript tracker library. You can choose to load the tracker from a CDN or host it on your own server. For production usage, we recommend [self-hosting the tracker](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/).

### Self-Hosted Library URL

This field is required if you choose to load the Snowplow JavaScript tracker from your own server. Enter the URL of the tracker library here.

> **Warning:** The default Tag doesn't have the permission to inject scripts from a custom URL.
>
> You will need to update the `Injects Scripts` permission to reflect the new location, by editing the `Snowplow Analytics v3/v4 Tag` template. Delete the content of the `Allowed URL Match Patterns` field, and type the full URL to the library there. Again, it must match what you input into the tag itself when creating it.
>
> ![The GTM Template Editor Permission tab with the \&quot;Injects scripts\&quot; permission expanded, showing an Allowed URL match patterns field containing a custom self-hosted tracker URL](/assets/images/modifying_permissions-d485193d56ce3212e8c83daea275800b.png)
>
> Modifying permissions **breaks the gallery link** and you will no longer be notified about updates to the template.
>
> ![A GTM warning dialog stating \&quot;Modified Community Gallery Templates will no longer receive updates provided by the developer\&quot; with Cancel and Continue buttons](/assets/images/modifying_breaks_gallery_link-7870229aea74cb186cc082ab6b9b2c36.png)

> **Note:** Since v1.1.0, an alternative to prevent breaking the gallery update link is to use the `Do not load` option from the corresponding drop down menu:
>
> ![The \&quot;Override JavaScript Library Host\&quot; option enabled with the Snowplow JavaScript Tracker Library dropdown set to \&quot;Do not load library\&quot;](/assets/images/host_drop_down_no_load-06e54c13559cb4115d3d5c4577540b8f.png)

### Library Version

This field is required if you choose to load the Snowplow JavaScript tracker from a CDN. Enter the version of the tracker library you want to load here.

## Application Settings

### Application ID

This is the unique identifier for your application. It is used to distinguish different applications in your Snowplow pipeline.

### Platform

This is the platform on which your application is running. This is used to distinguish different platforms in your Snowplow pipeline.

## Privacy Settings

### Respect 'Do Not Track'

This setting allows you to respect the Do Not Track setting in the user's browser. When enabled, the Snowplow JavaScript tracker will not track users who have enabled the Do Not Track setting in their browser, along with preventing any cookies from being set.

### Anonymous Tracking

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

#### Server Anonymisation

Server-side anonymisation affects user identifiers set server-side. In particular, these are the network\_userid property set in server-side cookie and the user IP address.

Setting the flag will add a `SP-Anonymous` HTTP header to requests sent to the Snowplow collector. The Snowplow pipeline will take care of anonymising the identifiers.

#### Anonymous Session Tracking

This setting disables client-side user identifiers but tracks session information. In practice, this means that events track the Session context entity but the userId property is a null UUID (00000000-0000-0000-0000-000000000000). In case Platform context is enabled, the IDFA identifiers will not be present.

#### Cookie Lifetime Extension Service

This allows you to set the endpoint for the [Cookie Lifetime Extension Service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/).

## Cookie Settings

### State Storage Strategy

This setting allows you to choose the strategy for storing the Snowplow tracker state. The available options are:

- `Cookie and Local Storage`: The Snowplow tracker will store the state in both cookies and local storage.
- `Cookie`: The Snowplow tracker will store the state in cookies only.
- `Local Storage`: The Snowplow tracker will store the state in local storage only.
- `None`: The Snowplow tracker will not store the state.

### Cookie Domain

This setting allows you to specify the domain for which the Snowplow tracker cookies will be set. This is useful when you want to track users across subdomains.

By default, `auto` will be used, which will set the domain to the root domain.

See [here](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-domain) for more information.

### Cookie Lifetime

This setting allows you to specify the lifetime of the Snowplow tracker cookies. By default, the cookies will be active for 2 years.

If the lifetime is set to `0`, the cookie will expire at the end of the session (when the browser closes). If set to `-1`, the first-party cookies will be disabled.

### Cookie SameSite

This setting allows you to specify the SameSite attribute for the Snowplow tracker cookies. The SameSite attribute is used to prevent CSRF attacks.

### Session Cookie Timeout

This setting allows you to specify the timeout for the session cookie. By default, the session cookie will expire after 30 minutes of inactivity.

### Synchronously Write Cookies

This setting allows you to specify whether the Snowplow tracker should [write cookies synchronously](/docs/sources/web-trackers/configuring-how-events-sent/#synchronous-cookie-writes). By default, the tracker will write cookies asynchronously.

## Dispatching

### Common

#### Dispatch Method

This setting allows you to choose the method for sending events to the Snowplow collector. The available options are:

- `POST`
- `GET`

It is recommended to use the default `POST` method for sending events to the collector as it allows for larger payloads, unless you have specific requirements for using the `GET` method.

#### Encode Into Base64

This setting allows you to encode the payload into Base64 before sending it to the collector. If you are using the `GET` method for sending events to the collector, it is recommended to enable this setting, as it will help prevent issues with special characters in the payload. Otherwise, it is recommended to leave this setting disabled to reduce the payload size.

#### Connection Timeout

This setting allows you to specify the timeout for the connection to the Snowplow collector. By default, the timeout is set to 5000 milliseconds.

### `POST` Specific

#### Buffer Size

This setting allows you to specify the number of events to buffer before sending them to the collector. By default, the buffer size is set to 1.

If you set the buffer size to a value greater than 1, the tracker will buffer events and send them in batches to the collector. Although this can help reduce the number of requests made to the collector, it comes at the expense of potential data loss for non-returning visitors.

#### POST Path

This setting allows you to specify the path to which the events will be sent. By default, the events will be sent to the `/com.snowplowanalytics.snowplow/tp2` path.

#### Maximum POST Payload Size

This setting allows you to specify the maximum size of the payload that will be sent to the collector. By default, the maximum payload size is set to 40000 bytes.

If an event is generated that is over the maximum payload size, the event will bypass the buffer and be sent immediately to the collector. This means that if it fails, it will not be retried.

#### Enable keepalive

This setting allows you to enable or disable the [keepalive](/docs/sources/web-trackers/configuring-how-events-sent/#keepalive-option-for-collector-requests) feature. This will enable requests to continue to be sent, even if the user navigates away from the page that sent the request.

Defaults to `false`.

## Predefined Contexts

Predefined contexts provide additional metadata for your Snowplow events. By including these contexts, you can capture common data points like device information, session details, or geolocation without having to define them manually.

Available predefined contexts are:

| Name                          | Description                                                                                                                              | Source Plugin                                                                                                |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `webPage`                     | Information about the web page where the event occurred.                                                                                 | [Web Page tracking](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity) |
| `gaCookies`                   | Information about the Google Analytics cookies.                                                                                          | [Google Analytics Cookies Plugin](/docs/sources/web-trackers/tracking-events/ga-cookies/)                    |
| `clientHints`                 | Information about the client's device.                                                                                                   | [Client Hints Plugin](/docs/sources/web-trackers/tracking-events/client-hints/)                              |
| `geolocation`                 | Information about the client's geolocation.                                                                                              | [Geolocation](/docs/events/ootb-data/geolocation/)                                                           |
| `session`                     | Information about the user session.                                                                                                      | [Session](/docs/events/ootb-data/user-and-session-identification/#session-entity)                            |
| `performanceNavigationTiming` | Retrieves data from the [PerformanceNavigationTiming](https://developer.mozilla.org/en-US/docs/Web/API/PerformanceNavigationTiming) API. | [Performance Navigation Timing](/docs/sources/web-trackers/tracking-events/timings/)                         |

---

# Snowplow GTM template
> Configure Snowplow v4 tag types in Google Tag Manager including ad tracking, button clicks, cart events, site search, timing, enhanced consent, ecommerce, error tracking, page views, link clicks, and form tracking with custom commands.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/snowplow-template/

This template implements the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) for Google Tag Manager. It allows for the sending of [Snowplow events](/docs/events/) from your website to your Snowplow collector.

Tag Types are the kinds of events that can be tracked with the Snowplow v4 Tag Template. Each tag type has its own set of options and parameters that can be configured.

You can also configure [plugins](/docs/sources/google-tag-manager/snowplow-template/plugins/) to use with this template.

## Ad Tracking

The Ad Tracking tag is used to track impressions and ad clicks. This can used by, for example, ad networks to identify which sites and web pages users visit across a network, so that they can be segmented.

**Ad Tracking Parameters**

All ad tracking events take the following common parameters:

| Name           | Required? | Description                                                            | Example |
| -------------- | --------- | ---------------------------------------------------------------------- | ------- |
| `advertiserId` | No        | The advertiser ID                                                      | 201     |
| `campaignId`   | No        | The campaign ID                                                        | 12      |
| `cost`         | No        | The cost of the ad                                                     | 5.5     |
| `costModel`    | No        | The cost model for the campaign. Must be one of `cpc`, `cpm`, or `cpa` | cpc     |

The ad tracking tag includes three event types, each with its own set of additional parameters:

### Impression Event

| Name           | Required? | Description                                                      | Example                   |
| -------------- | --------- | ---------------------------------------------------------------- | ------------------------- |
| `impressionId` | No        | Identifier for the particular impression instance                | 67965967893               |
| `targetUrl`    | No        | The destination URL                                              | <https://www.example.com> |
| `bannerId`     | No        | Adserver identifier for the ad banner (creative) being displayed | 23                        |
| `zoneId`       | No        | Adserver identifier for the zone where the ad banner is located  | 7                         |

### Click Event

| Name        | Required? | Description                                                      | Example                   |
| ----------- | --------- | ---------------------------------------------------------------- | ------------------------- |
| `targetUrl` | Yes       | The destination URL                                              | <https://www.example.com> |
| `clickId`   | No        | Identifier for the particular click instance                     | 12243253                  |
| `bannerId`  | No        | Adserver identifier for the ad banner (creative) being displayed | 23                        |
| `zoneId`    | No        | Adserver identifier for the zone where the ad banner is located  | 7                         |

### Conversion Event

| Name           | Required? | Description                                       | Example   |
| -------------- | --------- | ------------------------------------------------- | --------- |
| `conversionId` | No        | Identifier for the particular conversion instance | 743560297 |
| `category`     | No        | Conversion category                               | ecommerce |
| `action`       | No        | The type of user interaction                      | purchase  |
| `property`     | No        | Describes the object of the conversion            | shoes     |
| `initialValue` | No        | How much the conversion is initially worth        | 99        |

## Button Click Tracking

This tag will enable the tracking of clicks on buttons, covering both `<button>` and `<input type="button">` elements.

> **Info:** This does not track track a button click event directly, but rather will enable listen for button clicks.

### Filter

The `Filter` option allows you to specify a list of CSS classes that will be used to filter out elements that should not be tracked. This can be useful if you have a large number of buttons on your site, but only want to track clicks on a subset of them.

There are two types of filters you can use:

- Allow: Only elements that match the specified CSS classes will be tracked.
- Block: Elements that match the specified CSS classes will not be tracked.

Each filter takes a comma-separated list of CSS classes. For example, to only track clicks on buttons with the classes "track-me" and "click-me" you would add enter `track-me, click-me` into `Classes` when the `Allow` filter type is selected.

## Cart Tracking

The Cart Tracking tag is used to track interactions with a shopping cart. This can be used to track the addition and removal of items to a cart.

**Cart Tracking Parameters**

Add To Cart and Remove From Cart events take the following parameters:

| Name        | Required? | Description                                                     | Example    |
| ----------- | --------- | --------------------------------------------------------------- | ---------- |
| `sku`       | Yes       | The stock keeping unit (SKU) of the product being added/removed | 000345     |
| `unitPrice` | Yes       | The price of the product being added/removed                    | 12.99      |
| `quantity`  | Yes       | The quantity of the product being added/removed                 | 2          |
| `name`      | No        | The name of the product being added/removed                     | 'blue tie' |
| `category`  | No        | The category of the product being added/removed                 | 'clothing' |
| `currency`  | No        | The currency of the product being added/removed                 | 'USD'      |



> **Info:** We recommend migrating to the [Snowplow Ecommerce Tag](/docs/sources/google-tag-manager/ecommerce-tag-template/) template to take advantage of the latest ecommerce tracking features.

## Site Search

The Site Search tag can be used to track searches on your website, using the [Site Search Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/site_search/jsonschema/1-0-0)

**Site Search Parameters**

| Name           | Required? | Description                     | Example                  |
| -------------- | --------- | ------------------------------- | ------------------------ |
| `terms`        | Yes       | Search terms                    | \['unified, log']        |
| `filters`      | No        | Search filters                  | {'category': 'clothing'} |
| `totalResults` | No        | Results found                   | 10                       |
| `pageResults`  | No        | Results displayed on first page | 5                        |

## Timing

This is used to track user timing events, such as how long resources take to load.

**Timing Parameters**

| Name       | Required? | Description                    | Example            |
| ---------- | --------- | ------------------------------ | ------------------ |
| `category` | Yes       | Timing category                | 'load'             |
| `variable` | Yes       | Timed variable                 | 'map\_loaded'      |
| `timing`   | Yes       | Number of milliseconds elapsed | 50                 |
| `label`    | No        | Label for the event            | 'Map loading time' |

## Enhanced Consent

The Enhanced Consent tag is used to track user consent preferences. This can be used to track whether a user has given consent to certain tracking activities, such as tracking cookies or personalized ads.

**Enhanced Consent Events**

| Name           | Description                                             |
| -------------- | ------------------------------------------------------- |
| Allow All      | Track an acceptance of user consent.                    |
| Allow Selected | Track a specific selection of consented scopes.         |
| Pending        | Track the unconfirmed selection about user consent.     |
| Implicit       | Track the implicit consent on user consent preferences. |
| Deny           | Track a denial of user consent.                         |
| Expired        | Track the expiration of a consent selection.            |
| Withdrawn      | Track the withdrawal of user consent.                   |

**Enhanced Consent Parameters**

| Attribute            | Description                                                                         | Example                 |
| -------------------- | ----------------------------------------------------------------------------------- | ----------------------- |
| `basisForProcessing` | GDPR lawful basis for data collection & processing.                                 | Consent                 |
| `consentUrl`         | URI of the privacy policy related document.                                         | <https://example.com>   |
| `consentVersion`     | Version of the privacy policy related document.                                     | 1.0                     |
| `consentScopes`      | The scopes allowed after the user finalized their selection of consent preferences. | 'analytics, functional' |
| `domainsApplied`     | The domains for which this consent allows these preferences to persist to.          | <https://example.com>   |
| `gdprApplies`        | Determine if GDPR applies based on the user's geo-location.                         | -                       |

## Enhanced Ecommerce

For Google Analytics 4 compatible ecommerce tracking, see the dedicated [Snowplow v3 Ecommerce Tag](/docs/sources/google-tag-manager/previous-versions/v3/v3-tags/ecommerce-tag-template/) template instead.

> **Info:** The following Enhanced Ecommerce plugin has been deprecated in favor of the [Snowplow Ecommerce Tag](/docs/sources/google-tag-manager/ecommerce-tag-template/) template, it is recommended to use the new template for the latest ecommerce tracking features. This tag remains for compatibility with existing setups.

The Enhanced Ecommerce tag is used to track ecommerce events, such as product views, add to cart, and purchases.

When you select [Enhanced Ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/enhanced/), you are left with two options: Use Data Layer or Choose Variable. The way it works is very similar to Enhanced Ecommerce in Google Analytics.

If you choose the first, the template will look into the `dataLayer` structure for the most recently pushed [Enhanced Ecommerce object](https://www.simoahava.com/analytics/enhanced-ecommerce-guide-for-google-tag-manager/#data-types-actions), and map this object to the request to Snowplow Analytics.

If you selected `Choose Variable`, you need to provide a GTM variable that returns an object in the correct, expected format.

## Error Tracking

The Errors tracker tag provides a way to manually track any errors that occur on your website.

**Error Tracking Parameters**

| **Name**   | **Required?** | **Description**                     | **Example**              |
| ---------- | ------------- | ----------------------------------- | ------------------------ |
| `message`  | Yes           | Error message                       | 'Cannot get user object' |
| `filename` | No            | Filename or URL                     | 'shop.js'                |
| `lineno`   | No            | Line number of problem code chunk   | 23                       |
| `colno`    | No            | Column number of problem code chunk | 45                       |
| `error`    | No            | JS `ErrorEvent`                     | {Stack trace string}     |

## Page View

You can provide a `Custom Page Title` if you wish, and you can add a [custom entity](/docs/events/custom-events/#custom-entities) to the request, as also described [here](/docs/sources/web-trackers/tracking-events/page-views/).

If you enable `Page Activity Tracking`, page pings will be sent at intervals that you can specify.

The `Callback Function` is something you can set to a JavaScript function. If you set the callback, then instead of sending the page ping to Snowplow, the function gets invoked instead. See the [JavaScript tracker documentation](/docs/sources/web-trackers/tracking-events/activity-page-pings/#activity-tracking-callback) for more information.

## Link Click Tracking

The `Track Link Click` event is similar to regular parameter-based events, as it lets you add parameters and track a link click as a manually encoded hit.

The `Enable Automatic Link Click Tracking` adds listeners to the page, which will track clicks on links permitting they adhere to the allow/denylisted class names you can optionally provide. It is intended to trigger [early on a page](https://support.google.com/tagmanager/answer/7679319) to begin listening for click events, _not_ to trigger on click events like [Click Triggers](https://support.google.com/tagmanager/answer/7679320).

The `Fix Middle-click Tracking` adds a fix for some browsers where middle-clicks were not tracked properly.

If you check `Track HTML Content Of Clicked Link`, then the full text content of the link element will be sent to Snowplow as well.

> **Note:** This setting enables the tracker's native [Link Click Tracking plugin](/docs/sources/web-trackers/tracking-events/link-click/), it does not directly fire an event.

## Form Tracking

Form tracking has just two options, because filters and transformations won’t work with the custom template (due to lack of support for processing HTML elements). The options are to set form and/or field denylists and allowlists.

Denylists and allowlists for forms are a list of comma-separated HTML class names. If denylisted, then any form element with a listed class will not trigger the form event. If allowlisted, then _only_ form elements with a listed classname will be tracked.

For fields, denylists and allowlists work similarly, except they use the `name` attribute rather than the class.

Note that this tag enables the tracker's native [Form Tracking plugin](/docs/sources/web-trackers/tracking-events/form-tracking/), it does not directly fire an event. It is intended to trigger [early on a page](https://support.google.com/tagmanager/answer/7679319) to begin listening for form events, _not_ to trigger on form events like [Form Submit Triggers](https://support.google.com/tagmanager/answer/7679217).

## Custom commands

For any other commands which are supported by the Snowplow JavaScript Tracker v4, you can select the `[Custom Command]` option. Once selected, you can enter any function name and the associated parameter for that function. The parameter can either be a simple string, in examples such as `setUserId`:

![GTM tag Custom Command section with Command Name set to \&quot;setUserId\&quot; and Command Argument set to \&quot;user-123\&quot;](/assets/images/setUserId-7128fb7529a6fb0148cccf87e6ddebf4.png)

Or it can be set to a Custom JavaScript Variable in the instances where an Object should be passed to the function, such as with `enableAnonymousTracking`:

![GTM Custom JavaScript variable configuration named \&quot;Anonymous Tracking Options\&quot; containing a function that returns an options object with withServerAnonymisation set to true](/assets/images/enableAnonymousTracking_custom_variable-37f00a0b1fdc34f552bf62fb02e2983b.png)

And then use this variable as your Command Argument:

![GTM tag Custom Command section with Command Name set to \&quot;enableAnonymousTracking\&quot; and Command Argument set to \&quot;Anonymous Tracking Options\&quot;](/assets/images/enableAnonymousTracking_argument-2250e7ff142e56e8beb72b3674138c8b.png)

For a list of available commands, consult the [API documentation](https://github.com/snowplow/snowplow-javascript-tracker/blob/master/trackers/browser-tracker/src/api.ts) for the Browser Tracker

## Additional Tracking Parameters

![GTM tag Additional Tracking Parameters section showing a Context Entities dropdown for attaching custom context and a Set Custom Timestamp field with a UNIX milliseconds input](/assets/images/additional_tracking_parameters-7f4abc16eb1463fe636661c2141bf131.png)

### Add custom context entities

Using the Context Entities table allows you to attach [custom context entities](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-entity) to the Snowplow event. Each row should be set to a variable value that must be an array of custom context objects that will all be concatenated to add to the Event.

For example to manually attach the web page context, create a custom JavaScript variable that returns an array with the custom context object:

```js
function() {

    return [{

        schema: 'iglu:com.example/web_page/jsonschema/1-0-0',

        data: {

            id: '12345',

            title: 'Example Page',

            url: 'https://example.com'

        }

    }];

}
```

Then set this variable as the value in the Context Entities table.

### Set Custom Timestamp

You can also choose to [set the True Timestamp](/docs/sources/web-trackers/tracking-events/#custom-timestamp) with this field. The format must be UNIX time in milliseconds.

## Parameter Object

Tags that can derive their parameters from a Google Tag Manager variable are:

- [Ad Tracking](/docs/sources/web-trackers/tracking-events/ads/)
- [Cart Tracking](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/ecommerce/original/)
- [Error Tracking](/docs/sources/web-trackers/tracking-events/errors/)
- [Self-describing Event](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-event-self-describing)
- [Site Search](/docs/sources/web-trackers/tracking-events/site-search/)
- [Social Interaction](/docs/sources/web-trackers/tracking-events/social-media/)
- [Structured Event](/docs/fundamentals/canonical-event/#structured-events)
- [Timing](/docs/sources/web-trackers/tracking-events/timings/generic/)

You can set the Retrieve Parameters From Variable setting to a Google Tag Manager variable. This parameter _must_ return an object. In the object, the key-value pairs should reflect the named parameters in the [event documentation](/docs/sources/web-trackers/tracking-events/). For example, to have the variable populate an [Error event](/docs/sources/web-trackers/tracking-events/errors/), you could use a Custom JavaScript variable like this:

```javascript
function() {

  return {

    message: 'Some Error Happened',

    filename: 'somefile.js',

    lineno: 5,

    colno: 236,

    error: null

  }

}
```

Alternatively, you can set the drop-down to the value `No`, and add the parameters manually instead:

![GTM tag Parameter Configuration section showing Tag Type set to \&quot;Error Tracking\&quot;, Retrieve Parameters From Variable set to \&quot;No\&quot;, and a manually entered parameter row with Parameter Name \&quot;message (required)\&quot; and Parameter Value \&quot;Error Message\&quot;](/assets/images/adding_parameters_manually-c874b2064a6048631c221f1f3375a18e.png)

Some tag types will add additional selections to this section. Follow the [official tracker documentation](/docs/sources/web-trackers/tracking-events/) for more information about what each option does.

---

# Snowplow GTM plugins
> Load JavaScript tracker plugins in Google Tag Manager to extend tracking functionality. Configure external and inline custom plugins with CDN URLs or custom JavaScript variables for enhanced event tracking.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/snowplow-template/plugins/

Plugins are supported to provide an easy way to extend the functionality of the tracker. This is similar to the [JavaScript Tracker](/docs/sources/web-trackers/plugins/creating-your-own-plugins/) but with the caveat that plugins may be more limited in their functionality due to the constraints of the Google Tag Manager environment.

Each plugin can be loaded from external URLs or provided inline within GTM, and can optionally take some custom configuration. A plugin may add new methods to the tracker that can be called via [Custom Commands](/docs/sources/google-tag-manager/snowplow-template/#custom-commands).

> **Note:** Snowplow plugins are updated alongside the tracker version to ensure compatibility. For best results, keep the tracker and plugin versions in sync:
>
> - CDN Users: Use a GTM Variable to manage versions in URLs.
> - Self-Hosting: Update plugin files whenever you update the tracker version.

## `Load Plugins` table

Plugins are configured by using the `Load Plugins` table, within the Snowplow tag. Each row takes three values to load a plugin.

> **Tip:** Plugins are loaded by the tracker directly, not via the Tag Template: You do not need to adjust the Template permissions to allow loading plugins from these URLs.

### Plugin URL

The URL to load the plugin from, e.g. `https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-link-click-tracking@latest/dist/index.umd.min.js`

The tracker will try to load plugins each time they are requested. This is usually acceptable as the browser will have cached the first request; if you don't want this behavior we suggest creating an empty `Custom Command` Tag with no commands that loads the plugins; you can make this Tag fire [once per page](https://support.google.com/tagmanager/answer/6279951) and add it as a [Setup Tag](https://support.google.com/tagmanager/answer/6238868) to any other Tags that use its commands.

### Plugin Configuration

The name of the plugin to load. These values can be found on the respective plugin documentation.

For example, for the `Link Click Tracking` plugin, the values (`'snowplowLinkClickTracking,LinkClickTrackingPlugin'`) can be found in the snippet [here](/docs/sources/web-trackers/tracking-events/link-click/#toggle-link-click-tracking).

### Additional Configuration (optional)

Finally, the optional `Additional Configuration` field allows you to add any configuration the plugin may require.

There are two ways to format additional configuration:

- A single comma-seperated string value, which will be split into an array of strings and passed as arguments to the plugin. (e.g. `arg1,arg2,arg3`)
- A reference to a GTM Variable containing an Array of arguments to pass to the plugin.

Plugins are loaded in order, and processed before the configured `Tag Type` configuration is executed -- so you can load a plugin in the same Tag that uses its functionality via Custom Commands.

The plugins will remain configured in the tracker and be accessible to later Tags.

> **Warning:** The Tag Template will try to call `.indexOf(',')` on the `Additional Configuration` value, so values of types other than Array or String may fail and break the Tag. String values (before or after splitting) of `true`, `false`, `null`, and numeric values will become their respective typed JSON values. It is not possible to pass a single `null`, `undefined`, or empty string value as a parameter to a plugin, instead no arguments will be passed to the plugin.

## Custom plugins

[Inline plugins](/docs/sources/web-trackers/plugins/creating-your-own-plugins/#inline-plugins) are plugins that don't require being fetched from an external file to load.

You can create Inline plugins in GTM by using Custom JavaScript Variables in the `Plugin URL` field. The Variable should return an Object with a method that returns another Object meeting the [plugin interface](/docs/sources/web-trackers/plugins/creating-your-own-plugins/#plugin-interface) (any other methods on the outer Object will become tracker methods).

For `Plugin Configuration`, the UI enforces the comma-seperated values syntax required for external plugins and unconditionally calls `.split(',')` on the string. The tracker requires that for inline-plugins only a single string may be used. To work around this limitation, create another Custom JavaScript Variable that returns your constructor method name wrapped in an Object with a fake `split()` method:

```javascript
function() {

  return {

    split: function() {

      return "myInlineConstructorMethodName";

    }

  };

}
```

> **Info:** If your configuration includes functions, GTM will wrap those functions in [its sandbox](https://developers.google.com/tag-platform/tag-manager/templates/sandboxed-javascript), even when passed to the tracker for execution.
>
> Complex values like DOM elements will be replaced by `null` when passed to or returned from your function, which may make some plugins not function as intended.

---

# Snowplow sources overview
> Overview of Snowplow tracker SDKs for web, mobile, server-side platforms, and third-party webhook integrations to collect event data.
> Source: https://docs.snowplow.io/docs/sources/

Snowplow supports a wide range of sources which send events to your Collector endpoint.

Snowplow tracker SDKs are client- or server-side libraries that enable you to collect events from your own applications. For tracking events from third-party applications, use [webhooks](/docs/sources/webhooks/).

You can find the source code for each tracker on GitHub, where possible. The JavaScript tracker repository is a multirepository containing the web (JavaScript and Browser), Node.js, and React Native trackers.

All Snowplow trackers have open-source licenses (Apache 2.0 License, except for the JavaScript tracker which is BSD 3-Clause License).

## Client-side trackers

| Tracker                                                 | Language                     | Used in                                                 | [Snowtype](/docs/event-studio/implement-tracking/)? | Supported [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) | GitHub                                                                                     | Installed using                                                                                                        |
| ------------------------------------------------------- | ---------------------------- | ------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/) (JavaScript)         | TypeScript, JavaScript       | Web                                                     | ✅                                                   | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker)     | Script tag                                                                                                             |
| [Web](/docs/sources/web-trackers/) (Browser)            | TypeScript, JavaScript       | Web                                                     | ✅                                                   | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker)     | [npm](https://www.npmjs.com/package/@snowplow/browser-tracker), yarn, or pnpm                                          |
| [Google Tag Manager](/docs/sources/google-tag-manager/) | JavaScript                   | Web                                                     | ❌                                                   | Unified Digital, Attribution, Ecommerce                                        | n/a                                                                                        | [Tag Template Gallery](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-v4) |
| [Google AMP](/docs/sources/google-amp-tracker/)         | HTML                         | AMP HTML webpages                                       | ❌                                                   |                                                                                | n/a                                                                                        | Script tag                                                                                                             |
| [Pixel](/docs/sources/pixel-tracker/)                   | HTML                         | Email, web environments where JavaScript is unavailable | ❌                                                   |                                                                                | n/a                                                                                        | Script tag                                                                                                             |
| [Android](/docs/sources/mobile-trackers/)               | Kotlin, Java                 | Mobile                                                  | ✅                                                   | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-android-tracker](https://github.com/snowplow/snowplow-android-tracker)           | [Maven](https://mvnrepository.com/artifact/com.snowplowanalytics/snowplow-android-tracker)                             |
| [iOS](/docs/sources/mobile-trackers/)                   | Swift, Objective-C           | Mobile (iOS), macOS, tvOS, watchOS, visionOS            | ✅                                                   | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-ios-tracker](https://github.com/snowplow/snowplow-ios-tracker)                   | SPM or [Cocoapods](https://cocoapods.org/pods/SnowplowTracker)                                                         |
| [React Native](/docs/sources/react-native-tracker/)     | TypeScript, JavaScript       | Mobile                                                  | ✅                                                   | Unified Digital, Attribution                                                   | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker)     | [npm](https://www.npmjs.com/package/@snowplow/react-native-tracker)                                                    |
| [Flutter](/docs/sources/flutter-tracker/)               | Dart                         | Mobile                                                  | ✅                                                   | Unified Digital, Attribution, Media Player                                     | [snowplow-flutter-tracker](https://github.com/snowplow/snowplow-flutter-tracker)           | [pub.dev](https://pub.dev/packages/snowplow_tracker)                                                                   |
| [Roku](/docs/sources/roku-tracker/)                     | BrightScript, BrighterScript | Media player applications                               | ❌                                                   | Media Player                                                                   | [snowplow-roku-tracker](https://github.com/snowplow-incubator/snowplow-roku-tracker)       | [npm](https://www.npmjs.com/package/@snowplow/roku-tracker)                                                            |
| [WebView](/docs/sources/webview-tracker/)               | TypeScript, JavaScript       | Hybrid mobile applications                              | ❌                                                   | Unified Digital, Attribution                                                   | [snowplow-webview-tracker](https://github.com/snowplow-incubator/snowplow-webview-tracker) | [npm](https://www.npmjs.com/package/@snowplow/webview-tracker) or script tag                                           |

## Server-side trackers

| Tracker                                                                                                                              | Language                    | [Snowtype](/docs/event-studio/implement-tracking/)? | GitHub                                                                                 | Installed using                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [Node.js](/docs/sources/node-js-tracker/)                                                                                            | TypeScript, JavaScript      | ✅                                                   | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker) | [npm](https://www.npmjs.com/package/@snowplow/node-tracker)                                         |
| [Python](/docs/sources/python-tracker/)                                                                                              | Python                      | ❌                                                   | [snowplow-python-tracker](https://github.com/snowplow/snowplow-python-tracker)         | [PyPI](https://pypi.org/project/snowplow-tracker/)                                                  |
| [Golang](/docs/sources/golang-tracker/)                                                                                              | Go                          | ✅                                                   | [snowplow-golang-tracker](https://github.com/snowplow/snowplow-golang-tracker)         | [GitHub](https://github.com/snowplow/snowplow-golang-tracker/releases)                              |
| [Java](/docs/sources/java-tracker/)                                                                                                  | Java                        | ✅                                                   | [snowplow-java-tracker](https://github.com/snowplow/snowplow-java-tracker)             | [Maven](https://mvnrepository.com/artifact/com.snowplowanalytics/snowplow-java-tracker)             |
| [.NET](/docs/sources/net-tracker/)                                                                                                   | C#                          | ❌                                                   | [snowplow-dotnet-tracker](https://github.com/snowplow/snowplow-dotnet-tracker)         | [NuGet](https://www.nuget.org/packages/Snowplow.Tracker)                                            |
| [Unity](/docs/sources/unity-tracker/)                                                                                                | C#                          | ❌                                                   | [snowplow-unity-tracker](https://github.com/snowplow/snowplow-unity-tracker)           | [GitHub](https://github.com/snowplow/snowplow-unity-tracker/releases)                               |
| [C++](/docs/sources/c-tracker/)                                                                                                      | C++                         | ❌                                                   | [snowplow-cpp-tracker](https://github.com/snowplow/snowplow-cpp-tracker)               | CMake                                                                                               |
| [Scala](/docs/sources/scala-tracker/)                                                                                                | Scala                       | ❌                                                   | [snowplow-scala-tracker](https://github.com/snowplow/snowplow-scala-tracker)           | [Maven](https://mvnrepository.com/artifact/com.snowplowanalytics/snowplow-scala-tracker) or JCenter |
| [Ruby](/docs/sources/ruby-tracker/)                                                                                                  | Ruby                        | ❌                                                   | [snowplow-ruby-tracker](https://github.com/snowplow/snowplow-ruby-tracker)             | [RubyGems](https://rubygems.org/gems/snowplow-tracker/)                                             |
| [PHP](/docs/sources/php-tracker/)                                                                                                    | PHP                         | ❌                                                   | [snowplow-php-tracker](https://github.com/snowplow/snowplow-php-tracker)               | [Packagist](https://packagist.org/packages/snowplow/snowplow-tracker)                               |
| [Rust](/docs/sources/rust-tracker/)                                                                                                  | Rust                        | ❌                                                   | [snowplow-rust-tracker](https://github.com/snowplow/snowplow-rust-tracker)             | [crates.io](https://crates.io/crates/snowplow_tracker)                                              |
| [Lua](/docs/sources/lua-tracker/)                                                                                                    | Lua                         | ❌                                                   | [snowplow-lua-tracker](https://github.com/snowplow/snowplow-lua-tracker)               | [LuaRocks](https://luarocks.org/modules/snowplow/snowplowtracker)                                   |
| [Tracking CLI](/docs/sources/snowplow-tracking-cli/) (deprecated, see [Snowplow CLI](/docs/api-reference/snowplow-cli/send-events/)) | Command line, shell scripts | ❌                                                   | [snowplow-tracking-cli](https://github.com/snowplow/snowplow-tracking-cli/)            | [GitHub](https://github.com/snowplow/snowplow-tracking-cli/releases)                                |

## Webhooks

| Webhook                                          | Track                                                                |
| ------------------------------------------------ | -------------------------------------------------------------------- |
| [Adjust](/docs/sources/webhooks/adjust-webhook/) | Which marketing channels are driving mobile app installations        |
| [Iglu](/docs/sources/webhooks/iglu-webhook/)     | Any [Iglu](/docs/api-reference/iglu/)-compatible GET or POST request |
| [Iterable](/docs/sources/webhooks/iterable/)     | Events provided by Iterable                                          |
| [MailGun](/docs/sources/webhooks/mailgun/)       | Email activity logged by MailGun                                     |
| [Mandrill](/docs/sources/webhooks/mandrill/)     | Email activity logged by Mandrill                                    |
| [SendGrid](/docs/sources/webhooks/sendgrid/)     | Email activity logged by SendGrid                                    |
| [Zendesk](/docs/sources/webhooks/zendesk/)       | Events logged by Zendesk                                             |

---

# Configure how events are sent for the Java tracker
> Configure network connections, event batching, buffering, and retry logic for the Java tracker to optimize event delivery to your collector.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/configuring-how-events-are-sent/

A user interacts with your app: an event is generated and tracked using `Tracker.track()`. But the event must be sent to your event collector, to enter your pipeline, before it has any value.

The Java tracker allows configuration of the network connection, event sending, and buffering of events. All of these configurations are contained within the `NetworkConfiguration` and `EmitterConfiguration` classes. The default configurations will be sufficient for many Snowplow users. The tables below show the different configuration options you can set.

Using `NetworkConfiguration`:

| NetworkConfiguration | Description                | Required?                                      |
| -------------------- | -------------------------- | ---------------------------------------------- |
| collectorUrl         | Event collector endpoint   | Yes, unless an `HttpClientAdapter` is provided |
| httpClientAdapter    | `HttpClientAdapter` object | No                                             |
| cookieJar            | OkHttp `CookieJar` object  | No                                             |

The URL path for your collector endpoint should include the protocol, "http" or "https".

Using `EmitterConfiguration`:

| EmitterConfiguration      | Description                                | Required? | Default |
| ------------------------- | ------------------------------------------ | --------- | ------- |
| batchSize                 | Number of events to batch into one request | No        | 50      |
| bufferCapacity            | Maximum number of events buffered          | No        | 10 000  |
| eventStore                | `EventStore` object                        | No        |         |
| customRetryForStatusCodes | Map of HTTP status codes to retry or not   | No        |         |
| threadCount               | Number of threads to use                   | No        | 50      |
| requestExecutorService    | `ScheduledExecutorService` object          | No        |         |
| callback                  | `EmitterCallback` object                   | No        |         |

See the API docs for the full [NetworkConfiguration](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/configuration/NetworkConfiguration.html) and [EmitterConfiguration](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/configuration/EmitterConfiguration.html) details.

The default `Emitter` is the `BatchEmitter`, which sends events asynchronously in batches using POST requests. If you need to send events synchronously, or with GET requests, we have provided an `Emitter` interface so you can create your own.

To create a `Tracker` manually, start by initialising an `Emitter`. The simplest `BatchEmitter` initialisation looks like this:

```java
BatchEmitter emitter = new BatchEmitter(new NetworkConfiguration("http://collectorEndpoint"));
```

When an `Event` is tracked using `Tracker.track()`, a payload (`TrackerPayload`) is generated from the `Event`. The payload is added to the `BatchEmitter`'s `InMemoryEventStore` buffer. This triggers a check on the size of the buffer. The number of stored events is compared with the configured `batchSize`; the default is 50 events per batch. If there are enough events, a batch's worth is asynchronously removed from the buffer for sending. The `BatchEmitter` prepares a request payload containing all the event payloads, and attempts to send it. On receiving a successful HTTP response code (2xx), the events are considered sent, and permanently deleted from the buffer.

If the event buffer is full, the payload will be dropped. Priority is given to older events. In this case, `Tracker.track()` returns `null` rather than the payload `eventId`.

### What happens if an event fails to send?

After a sending attempt, the fate of an event depends on which status code was received. A 2xx code is always considered successful. If the `HttpClientAdapter` returns a failure code (anything other than 2xx, with certain exceptions, see below), the events (as `TrackerPayload` objects) are returned to the buffer. They will be retried in future sending attempts. The returned events are added to the start of the buffer queue, as older events are prioritised over newer ones.

To prevent unnecessary requests being made while the collector is unavailable, an exponential backoff is added to all subsequent event sending attempts. This resets after a request is successful. The maximum backoff time between attempts is 10 minutes.

The status codes `400 Bad Request`, `401 Unauthorised`, `403 Forbidden`, `410 Gone`, or `422 Unprocessable Entity` are the exceptions: they are not retried by default. Payloads in requests receiving these responses are not returned to the buffer for retry. They are just deleted.

Configure which codes to retry on or not using the `EmitterConfiguration` `customRetryForStatusCodes()` method when creating your tracker. This method takes a map of status codes and booleans (true for retry and false for not retry):

```java
// by default 403 isn't retried, but 500 is

Map<Integer, Boolean> customRetry = new HashMap<>();

customRetry.put(403, true);

customRetry.put(500, false);



Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration("https://collector"),

                new EmitterConfiguration().customRetryForStatusCodes(customCodes));



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().customRetryForStatusCodes(customRetry));
```

See below for information about retries within HTTP clients.

### Configuring how many events to send in one request

The BatchEmitter sends events in batches. This is more efficient than sending event requests singly, as only one set of POST headers is required for a number of events. The event collector receives the request and separates out the individual event payloads.

The default batch size is 50 events. If you have a high event volume, larger batches may be more suitable. However, there is theoretically a risk of creating oversized requests, which could result in `413 Payload Too Large` responses from your collector. The Java tracker and Snowplow collector do not currently have the ability to split oversized requests into smaller ones.

Configure the batch size like this:

```java
Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration("https://collector"),

                new EmitterConfiguration().batchSize(100));



// Batch size can be updated after initialization

tracker.getEmitter().setBatchSize(10)



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().batchSize(100))
```

The `batchSize` property was called `bufferSize` in versions before 0.12.

To force send all buffered events, as a single request:

```java
emitter.flushBuffer()
```

### Configuring how events are buffered

The Java tracker sends events asynchronously: the application is not blocked waiting for the tracked event to be sent. The tracked events are stored in memory until there are enough to send, or while the network is down. The default event store is `InMemoryEventStore`, added in version 0.12. This class stores the payloads in a queue, specifically a `LinkedBlockingDeque`.

The default buffer capacity is 10 000 events. This is the number of events that can be stored. When the buffer is full, new tracked payloads are dropped, so choosing the right capacity is important. If events are accumulating in your buffer because of a very high event volume, rather than because of network outages, consider using more threads (see below).

The theoretical maximum capacity is that of a`LinkedBlockingDeque`: `Integer.MAX_VALUE`. It's likely your application would run out of memory before buffering that many events.

Creating a `BatchEmitter` with a custom maximum buffer capacity:

```java
Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration("https://collector-endpoint"),

                new EmitterConfiguration().bufferCapacity(100000));



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().bufferCapacity(100000));
```

The `BatchEmitter` in this tracker will store 100 000 events before starting to lose data.

We also provide an `EventStore` interface. To use a custom EventStore:

```java
Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration("https://collector-endpoint"),

                new EmitterConfiguration().eventStore(EventStore));



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().eventStore(EventStore));
```

If `bufferCapacity` is provided at the same time as `eventStore`, the `bufferCapacity` value will be discarded.

### Configuring the network connection

We currently offer two different HTTP clients that can be used to send events to your collector: OkHttp or Apache HTTP. Both libraries have broadly the same features, with some differences in their default configurations. If neither of these HTTP clients is suitable, we also provide an `HttpClientAdapter` interface. The `HttpClientAdapter` is a wrapper for HTTP client objects.

> **Note:** Gradle users: [different dependencies](/docs/sources/java-tracker/installation-and-set-up/) can be configured if you are using OkHttp or Apache HTTP.

By default, the Java tracker uses OkHttp; an `OkHttpClientAdapter` object is generated when a `BatchEmitter` is created. See below for how to use Apache HTTP instead, or how to customise the OkHTTP setup.

To specify a completely different client adapter, use the `HttpClientAdapter` interface and initialize the `Tracker` like this:

```java
HttpClientAdapter adapter = {{ your implementation here }}



Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration(adapter));
```

Note that `collectorUrl` is not a required parameter for `NetworkConfiguration` when an `HttpClientAdapter` is specified. The `collectorUrl` is normally used to create the default `OkHttpClientAdapter`, therefore if a URL was provided here, it would be ignored.

HTTP request retry can be configured within the HTTP clients, on top of the Java tracker's handling of unsuccessful requests. The default HTTP client, OkHttp, [retries after certain types of connection failure](https://square.github.io/okhttp/features/calls/#retrying-requests) by default. The Apache HTTP Client [retries a request up to 3 times](https://www.javadoc.io/doc/org.apache.httpcomponents/httpclient/4.3.5/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.html) by default.

#### OkHttpClient

The simplest OkHttpClient initialization looks like this:

```java
OkHttpClient client = new OkHttpClient();
```

This is the default as used in the `BatchEmitter`.

To add configuration, pass a `OkHttpClientAdapter` during Tracker initialization. For example, setting timeouts:

```java
OkHttpClient client = new OkHttpClient.Builder()

      .connectTimeout(5, TimeUnit.SECONDS)

      .readTimeout(5, TimeUnit.SECONDS)

      .writeTimeout(5, TimeUnit.SECONDS)

      .build();



OkHttpClientAdapter adapter = new OkHttpClientAdapter(

      "http://collector-endpoint.com",

      client

);



Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration(adapter));
```

The URL is the address for your collector. See [Square's API docs](https://square.github.io/okhttp/3.x/okhttp/okhttp3/OkHttpClient.Builder.html) for the full list of options.

#### Apache HTTP Client

The simplest Apache HTTP Client initialization looks like this:

```java
CloseableHttpClient client = HttpClients.createDefault();
```

Wrap the Client in an `ApacheHttpClientAdapter` to pass it to the tracker during initialization.

You are encouraged to research how best to set up your Apache Client for maximum performance. For example, by default the Apache Client will never time out, and will also allow only two outbound connections at a time. In this code block, a `PoolingHttpClientConnectionManager` is used to allow up to 50 concurrent outbound connections:

```java
PoolingHttpClientConnectionManager manager = new PoolingHttpClientConnectionManager();

manager.setDefaultMaxPerRoute(50);



CloseableHttpClient client = HttpClients.custom()

      .setConnectionManager(manager)

      .build();



ApacheHttpClientAdapter adapter = new ApacheHttpClientAdapter(

      "http://collector-endpoint.com",

      client

);



Tracker tracker = Snowplow.createTracker(

                new TrackerConfiguration("namespace", "appId"),

                new NetworkConfiguration(adapter));
```

The URL is the address for your collector. See [Apache's HttpClient docs](https://hc.apache.org/httpcomponents-client-4.5.x/index.html) for more information about configuring the client.

### Configuring the Java tracker threads

The `BatchEmitter` contains threads for concurrent event sending. This is managed by an `ScheduledThreadPoolExecutor`. By default, the thread pool has up to 50 threads, helpfully named e.g. "snowplow-emitter-pool-1-request-thread-1". The bigger the pool of threads, the faster events can be sent. Set the number of threads depending on many events you are sending, and how strong a computer the tracker is running on.

The process of getting events from the buffer, creating a request payload, and sending the POST request occurs within a single thread.

Specifying the maximum number of event sending threads, in this case to 1:

```java
Tracker tracker = Snowplow.createTracker(

                  new TrackerConfiguration("namespace", "appId"),

                  new NetworkConfiguration("https://collector"),

                  new EmitterConfiguration().threadCount(1));



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().threadCount(1));
```

It's also possible to provide your own `ScheduledExecutorService`:

```java
Tracker tracker = Snowplow.createTracker(

                  new TrackerConfiguration("namespace", "appId"),

                  new NetworkConfiguration("https://collector"),

                  new EmitterConfiguration().requestExecutorService(ScheduledExecutorService));



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().requestExecutorService(ScheduledExecutorService));
```

The default thread pool uses non-daemon threads. To stop the threads and shut down the `ExecutorService`, call `Emitter.close()` (or `Tracker.close()`). There is no way to restart the `Emitter` after this.

### Persisting cookies using a CookieJar

> **Note:** The `OkHttpClientWithCookieJarAdapter` was released in the version 2.0.0 of the Java tracker.

As described [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/), the event collector sets a third-party cookie. This cookie is extracted during event processing (enrichment phase) into the `network_userid` property, the server-side user identifier. To persist this cookie across requests, use the `OkHttpClientWithCookieJarAdapter` when creating your `BatchEmitter` or `Tracker`. Note that the `OkHttpClientWithCookieJarAdapter` uses an in-memory cookie jar, so the cookies, and `network_userid`, will not persist when it goes out of memory.

The simplest implementation looks like this:

```java
Tracker tracker = Snowplow.createTracker(

                  new TrackerConfiguration("namespace", "appId"),

                  new NetworkConfiguration(new OkHttpClientWithCookieJarAdapter("http://collector")));



// A BatchEmitter can also be created directly

BatchEmitter emitter = new BatchEmitter(networkConfig);
```

The specified `CookieJar` will be ignored if a custom `HttpClientAdapter` is provided. To use a `CookieJar` with a custom `OkHttpClientAdapter`, it must be added using the `OkHttpClient.Builder`:

```java
OkHttpClient httpClient = new OkHttpClient.Builder()

                  .cookieJar(new CollectorCookieJar())

                  .build();

adapter = new OkHttpClientAdapter("http://collectorEndpoint", httpClient);



Tracker tracker = Snowplow.createTracker(

                  new TrackerConfiguration("namespace", "appId"),

                  new NetworkConfiguration(adapter));
```

Any custom `OkHttp` `CookieJar` (other than the `CollectorCookieJar` provided in the tracker) could be used instead.

### Using the Emitter callback

To gain visibility on tracker activity, you may wish to take advantage of the `EmitterCallback` interface. This interface was added in v1.0.0, and requires `onSuccess()` and `onFailure()` methods.You can use `EmitterCallback` to create your own tracker metrics.

The `onSuccess()` callback is called when a request has successfully sent to the event collector (HTTP status code 2xx). It takes a list of the `TrackerPayload` objects that were batched into the request.

The `onFailure()` callback is called at several different points of failure. It also takes a list of the `TrackerPayload` objects involved, however, two other parameters are required. The first is the enum `FailureType`, explaining what kind of failure has occurred. The second is a boolean for whether or not the events will be returned to the buffer for sending retry.

The possible `FailureType` options are: `REJECTED_BY_COLLECTOR`, when HTTP status codes other than 2xx are received for a request; `TRACKER_STORAGE_FULL`, when events are lost because the `InMemoryEventStore` buffer is full; `HTTP_CONNECTION_FAILURE`, for unsuccessful requests or `HttpClientAdapter` exceptions; or `EMITTER_REQUEST_FAILURE`, for a `BatchEmitter` exception.

Configure a callback in your tracker like this:

```java
EmitterCallback callback = {{ your implementation here }}

Tracker tracker = Snowplow.createTracker(

                  new TrackerConfiguration("namespace", "appId"),

                  new NetworkConfiguration("https://collector"),

                  new EmitterConfiguration().callback(callback));
```

---

# Custom tracking with the Java tracker
> Track custom events and add context entities using self-describing JSON schemas with the Java tracker to capture business-specific data.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/custom-tracking-using-schemas/

Self-describing (self-referential) JSON schemas are at the core of Snowplow tracking. Read more about them [here](/docs/fundamentals/schemas/). They allow you to track completely customised data, and are also used internally throughout Snowplow pipelines.

In all our trackers, self-describing JSON are used in two places. One is in the `SelfDescribing` event type that wraps custom self-describing JSONs for sending. The second use is to attach custom data to any tracked event. It's one of the most powerful Snowplow features.

When tracking user behavior, the event describes the specific activity they performed, e.g. a user added an item to an eCommerce cart. To understand the meaning of the event, and how it relates to your business, it's ideal to also track the relatively persistent environment in which the activity was performed. For example, is the user a repeat customer? Which item did they add, and how many are in stock?

These environmental factors can be tracked as the event "context", using self-describing JSON. When self-describing JSON are tracked as part of an event, they are called "entities". All the entities of an event together form the context. Read more in this [thorough blog post](https://snowplowanalytics.com/blog/2020/03/25/what-are-snowplow-events-and-entities-and-what-makes-them-so-powerful/).

### Adding custom entities to any event

Every `Event.Builder` in the Java tracker allows for a list of `SelfDescribingJson` objects to be added to the `Event`. It's fine to add multiple entities of the same type. There's no official limit to how many entities you can add to a single event, but consider if the payload size could become problematic if you are adding a large number.

Context entities can be added to any event using the `customContext()` Builder method:

```java
// This example shows an SelfDescribing event, but all events can have context

SelfDescribing selfDescribing = SelfDescribing.builder()

            .eventData(dataAsSelfDescribingJson)

            .customContext(listOfEntitiesAsSelfDescribingJson)

            .build();
```

Event context is sent as a JSON inside the event payload. During [enrichment](/docs/pipeline/enrichments/), it is separated into columns for each different schema used.

A note on nomenclature: entities were originally called "context", with the context called "contexts". The old nomenclature is still used in some parts of Snowplow, meaning that enriched events in the data warehouse refer to "contexts"/"context" rather than "context"/"entity".

### Global context

The Java tracker does not yet provide the ability to automatically assign entities to specific events.

### Self-describing JSON in the Java tracker

The Java tracker provides the `SelfDescribingJson` class for custom events and entities. There is no in-built distinction between schemas used for events and those used for entities: they can be used interchangably.

Your schemas must be accessible to your pipeline, within an [Iglu server](/docs/api-reference/iglu/). Tracked events containing self-describing JSON are validated against their schemas during the enrichment phase of the pipeline. If the data don't match the schema, the events end up as [failed events](/docs/fundamentals/failed-events/).

A self-describing JSON needs two keys, `schema` and `data`. The `schema` key is the Iglu URI for the schema. The `data` value must match the properties described by the specified schema. It is usually provided as a map.

A simple initialisation looks like this:

```java
// This map will be used for the "data" key

Map<String, String> eventData = new HashMap<>();

eventData.put("targetUrl", "https://www.snowplowanalytics.com")



// This is the Iglu schema URI

String schema_uri = "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1"



// Wrap the map in a SelfDescribingJson

// The specified schema allows for a String property "targetUrl"

SelfDescribingJson sdj = new SelfDescribingJson(schema_uri, eventData);
```

See the API docs for the full [SelfDescribingJson](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/payload/SelfDescribingJson.html) constructor options.

JSON can be sent base-64 encoded or not. By default, they are encoded. This is set as part of the [Tracker configuration](/docs/sources/java-tracker/installation-and-set-up/#configuring-the-tracker).

---

# Example applications for the Java tracker
> Reference implementations and sample code to help you get started with the Java tracker.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/demos/

Check out the following example applications that can be used as a reference for how to use the tracker, or as a starting point for your own implementation.

- [Simple Console](https://github.com/snowplow/snowplow-java-tracker/tree/master/examples/simple-console)
- [Benchmarking](https://github.com/snowplow/snowplow-java-tracker/tree/master/examples/benchmarking)

---

# Java tracker SDK
> Add event tracking to Java-based desktop apps, server applications, servlets, and games with the Snowplow Java tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/

The Snowplow Java Tracker lets you add analytics to your [Java](http://www.java.com/en/)-based desktop and server apps, servlets and games.

---

# Install the Java tracker
> Install and configure the Java tracker using Maven, Gradle, or direct download for JDK8+ applications.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/installation-and-set-up/

The Snowplow Java tracker has been built and tested using Java versions 8, 11 and 17, so should work within any Java application built using JDK8 upwards. The Java tracker is also usable from Scala.

The current tracker version is 2.1.0. New issues and pull requests are very welcome! Find the Github repository [here.](https://github.com/snowplow/snowplow-java-tracker)

## Installing

### Install using Maven

Add into your project’s `pom.xml`:

```xml
<dependency>

  <groupId>com.snowplowanalytics</groupId>

  <artifactId>snowplow-java-tracker</artifactId>

  <version>2.1.0</version>

</dependency>
```

### Install using Gradle

From version 0.10.1 onwards, we provide out-of-the-box support for sending events via OkHttp or Apache HTTP. The appropriate dependencies must be specified. The default tracker configuration uses OkHttp.

Add this into your project’s `build.gradle` for the default installation with OkHttp support:

```gradle
dependencies {

  implementation 'com.snowplowanalytics:snowplow-java-tracker:2.1.0'

  implementation ('com.snowplowanalytics:snowplow-java-tracker:2.1.0') {

      capabilities {

          requireCapability 'com.snowplowanalytics:snowplow-java-tracker-okhttp-support'

      }

  }

}
```

Adding dependencies for Apache HTTP support instead (read [this page](/docs/sources/java-tracker/configuring-how-events-are-sent/) for how to configure this):

```gradle
dependencies {

  implementation 'com.snowplowanalytics:snowplow-java-tracker:2.1.0'

  implementation ('com.snowplowanalytics:snowplow-java-tracker:2.1.0') {

      capabilities {

          requireCapability 'com.snowplowanalytics:snowplow-java-tracker-apachehttp-support'

      }

  }

}
```

If you are using your own `HttpClientAdapter` implementation and/or will be installing dependencies separately:

```gradle
dependencies {

  implementation 'com.snowplowanalytics:snowplow-java-tracker:2.1.0'

}
```

### Install by direct download

You can also manually insert the tracker by downloading the jar directly from [Maven Central](https://search.maven.org/search?q=a:snowplow-java-tracker).

### Install in Scala project (SBT)

The Snowplow Java tracker is also usable from Scala. Add this to your SBT config:

```scala
// Dependency

val snowplowTracker = "com.snowplowanalytics"  % "snowplow-java-tracker"  % "2.1.0"
```

## Setting up

The simplest initialization looks like this:

```java
import com.snowplowanalytics.snowplow.tracker.Snowplow;

import com.snowplowanalytics.snowplow.tracker.Tracker;



Tracker tracker = Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");
```

The URL path for your collector endpoint should include the protocol, "http" or "https". The Java tracker is able to send events to either.

The `Snowplow` interface, added in v1, contains static methods to help initialise and manage `Tracker` objects. This is especially useful when multiple trackers are needed (see [here](/docs/sources/java-tracker/using-multiple-trackers/)). See the API docs for the full [Snowplow](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Snowplow.html) details.

The [Java tracker Github repository](https://github.com/snowplow/snowplow-java-tracker) includes a mini demo, "simple-console". Follow the instructions in the README to send one event of each type to your event collector. Simple-console is provided as a simple reference app to help you set up the tracker.

These are the required objects for tracking using the Java tracker:

| Class                 | Function               |
| --------------------- | ---------------------- |
| `Tracker`             | Tracks events          |
| subclasses of `Event` | What you want to track |

### Configuring the `Tracker`

The `Tracker` class has the responsibility for tracking [events](/docs/sources/java-tracker/tracking-events/). Certain properties can or must also be set when creating a `Tracker`, which will be attached to all events. These are `trackerNamespace`, `appId`, and `platform`.

Both `trackerNamespace` and `appId` are required arguments when creating a `Tracker`. Snowplow events are designed to be stored in a single data warehouse/lake, regardless of their source, to make data modeling easier and provide a single valuable source of truth for your business. The tracker namespace allows you to distinguish events sent by this specific `Tracker`, if you are using multiple `Tracker` instances within your app. It's also the identifier for `Tracker` objects in the `Snowplow` class. The `appId` allows you to identify events from this specific application, if you are tracking from multiple places.

The other Tracker property that will be added to all tracked events is `platform`. This is set by default to `srv` - "server-side app". To set another valid platform type, use the `DevicePlatform` enum during construction.

The final two `Tracker` configuration options are whether to use base-64 encoding, and whether to add a `Subject` object (see [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/) for details about `Subject`). By default, JSONs within the event are sent base-64 encoded. This can be set to `false` here at `Tracker` initialization.

To create a `Tracker` with custom configuration, use the `TrackerConfiguration` class.

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("namespace", "appId")

        .base64Encoded(false)

        .platform(DevicePlatform.Desktop);



// Use the Snowplow class to create Trackers

Tracker tracker = Snowplow.createTracker(

    trackerConfig,

    new NetworkConfiguration("http://collector"));



// Alternatively, create an Emitter first, and then create a Tracker directly

// Emitters are introduced in the next section

BatchEmitter emitter = new BatchEmitter(new NetworkConfiguration("http://collector"));

Tracker tracker = new Tracker(trackerConfig, emitter);
```

This `Tracker` will produce events with the `platform` value `pc`.

See the API docs for the full [Tracker](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Tracker.html), [TrackerConfiguration](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/configuration/TrackerConfiguration.html) and [Snowplow](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Snowplow.html) details.

### Stopping the tracker

The default `Emitter`, `BatchEmitter`, contains a pool of non-daemon threads managed by a `ScheduledExecutorService`. To close the executor service and stop the threads, use the `close()` method. This method can be called directly on an `Emitter`, or via a `Tracker`:

```java
tracker.close()



// It's also possible to close the Emitter directly

tracker.getEmitter().close()

emitter.close()
```

There is no way to restart the `Emitter` after this. Note that if your app is not quit, events can still be tracked after `close()` has been called: they will accumulate in the `Emitter` buffer, unable to be sent.

### Logging

Logging in the Tracker is done using SLF4J. The majority of the logging set as `DEBUG` so it will not overly populate your own logging.

Since Java tracker v0.11, user-supplied values are only logged at `DEBUG` level.

---

# Migration guides for the Java tracker
> Upgrade guides to help you migrate between major versions of the Java tracker.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/migration-guides/

This section contains information to help you upgrade to newer versions of the Java tracker.

---

# From version 0.11 to 0.12
> Migration guide for upgrading the Java tracker from version 0.11 to 0.12 with breaking changes and new features.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/migration-guides/migration-guide-v0-12/

Several new features were added in v0.12. See the [Github](https://github.com/snowplow/snowplow-java-tracker) changelog for full details. Unfortunately, these improvements involve multiple breaking changes. See below for a guide to the changes.

### `BatchEmitter`

The `BatchEmitter.Builder` configuration options have changed.

**Updated**:

- `bufferSize` has been renamed `batchSize`.
- `requestExecutorService()` now requires a `ScheduledExecutorService`, rather than just `ExecutorService`.

**Removed**:

- `requestCallback()` is gone, since events that fail to send will now be automatically retried.

**Added** (these are not breaking changes):

- `eventStore()`. It's now possible to design your own event buffer using the `EventStore` interface. This is an optional setting, with a default of `InMemoryEventStore`.
- `bufferCapacity()`. Set the maximum capacity of the default `InMemoryEventStore`. Again, this setting is optional. The default is `Integer.MAX_VALUE` (the maximum capacity of the queue used).

**Example**: creating a BatchEmitter with the relevant options\
Old API:

```java
BatchEmitter batchEmitter = BatchEmitter.builder()

        .url("http://collector.url")

        .bufferSize(20)

        .requestCallback( {{ A callback }} )

        .requestExecutorService(ExecutorService)

        .build();
```

Version 0.12:

```java
BatchEmitter batchEmitter = BatchEmitter.builder()

        .url("http://collector.url")

        .batchSize(20)

        .eventStore(EventStore)

        .bufferCapacity(1000000) // this won't do anything since eventStore is specified

        .requestExecutorService(ScheduledExecutorService)

        .build();
```

### `SimpleEmitter`

`SimpleEmitter` has been deprecated. Please use `BatchEmitter` instead, or create your own `Emitter` using the provided interface. Like `SimpleEmitter`, the `BatchEmitter` sends events asynchronously. However, requests are made using POST, rather than GET. We strongly recommend sending events in batches, but to mimic SimpleEmitter in sending events one-by-one, use a `batchSize` of 1.

**Example**: replacing `SimpleEmitter` with `BatchEmitter`\
Old API:

```java
SimpleEmitter simpleEmitter = SimpleEmitter.builder()

        .url("http://collector.url")

        .build();



Tracker tracker = new Tracker.TrackerBuilder(simpleEmitter, "namespace", "appId")

        .build();
```

Version 0.12:

```java
BatchEmitter batchEmitter = BatchEmitter.builder()

        .url("http://collector.url")

        .batchSize(1)

        .build();



Tracker tracker = new Tracker.TrackerBuilder(batchEmitter, "namespace", "appId")

        .build();
```

### `Emitter` interface

Every method in the `Emitter` interface has been updated!

`setBufferSize()` and `getBufferSize()` have been renamed `setBatchSize()` and `getBatchSize()`.

`getBuffer()` now returns a list of `TrackerPayload` objects rather than `TrackerEvent` objects (which no longer exist).

Finally, `emit()` is now more accurately called `add()`, as in "add to buffer". It takes a `TrackerPayload` object, not a `TrackerEvent`, and has a new return type: a `boolean`, used to confirm that the `TrackerPayload` has been successfully added to the buffer.

### Getting an event's `eventId`

We are aware of some use cases involving exporting the `eventId`s of tracked Events to third-party apps. The `eventId` is now returned from `Tracker.track()`. It's returned in a list to allow for `EcommerceTransaction` events, which generate multiple payloads. If the event buffer is full, the event is lost. In this case, `null` will be returned instead of the `eventId`.

**Example**: getting the `eventId`\
Old API:

```java
PageView pageView = PageView.builder()

        .pageUrl("https://www.snowplowanalytics.com")

        .build();



String eventId = pageView.getEventId();
```

Version 0.12:

```java
PageView pageView = PageView.builder()

        .pageUrl("https://www.snowplowanalytics.com")

        .build();



List<String> eventIds = tracker.track(pageView);

String eventId = eventIds.get(0);
```

### `Event`, `AbstractEvent`, and child classes

Several methods have been removed: the `Event` interface and `AbstractEvent` methods `getDeviceCreatedTimestamp()`, and `getEventId()`; and the `AbstractEvent.Builder` methods `deviceCreatedTimestamp()`, and `eventId()`. The deprecated `timestamp()` and `getTimestamp()` methods have been removed from these classes too.

Having discovered them, we deleted these methods immediately (rather than deprecating) as we considered them very dangerous. Allowing custom UUIDs can accidentally lead to non-unique "unique" identifiers, which causes big problems for pipelines, and risks data loss.

Despite the name, `Event` objects are no longer associated with an `eventId` ; this is generated when the `Payload` object is made. The main purpose of the `eventId` is to provide a UUID for events once they have been received by the collector and are in the pipeline.

**Example**: creating a PageView with all the options\
Old API:

```java
PageView pageViewEvent = PageView.builder()

        .pageTitle("Snowplow Analytics")

        .pageUrl("https://www.snowplowanalytics.com")

        .referrer("https://www.google.com")

        .customContext(SelfDescribingJson)

        .subject(Subject)

        .trueTimestamp(1646834667343L)

        .deviceCreatedtimestamp(1646834667123L)

        .eventId("UUID")

        .build();
```

Version 0.12:

```java
PageView pageViewEvent = PageView.builder()

        .pageTitle("Snowplow Analytics")

        .pageUrl("https://www.snowplowanalytics.com")

        .referrer("https://www.google.com")

        .customContext(SelfDescribingJson)

        .subject(Subject)

        .trueTimestamp(1646834667343L)

        .build();
```

### `TrackerEvent` and callbacks

This class no longer exists. It was a wrapper around tracked `Event` objects to allow request callbacks. `Event`s were stored in the buffer as `TrackerEvent` until event sending, when a `TrackerPayload` would be extracted. `TrackerPayload` objects are now stored directly.

Callbacks allowed developers to re-track events that failed to send - if a HTTP response code other than 2xx was received. This put the burden on users to handle retry. Now that the tracker automatically retries, callbacks are no longer necessary, and have been removed.

---

# From version 0.12 to 1.0
> Migration guide for upgrading the Java tracker from version 0.12 to 1.0 with API improvements and configuration changes.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/migration-guides/migration-guide-v1/

We've made multiple improvements to the API, and to the configuration, for Java tracker v1. See the [Github changelog](https://github.com/snowplow/snowplow-java-tracker/releases/tag/1.0.0) for full details. See below for a guide to the changes.

## API improvements (breaking changes)

### `Snowplow` interface, `Configuration` classes, and deprecated Builders

This is a big change. We wanted an intuitive API going into v1, so we've added a `Snowplow` interface to easily create and manage trackers. We've added Configuration classes - `TrackerConfiguration`, `NetworkConfiguration`, `EmitterConfiguration` and `SubjectConfiguration` - and several new constructors for instantiating the tracker or tracker components. As part of this work, we've deprecated the builders for the `Tracker`, `BatchEmitter`, `HttpClientAdapter` and `Subject` classes.

Instantiating a tracker with default configuration only requires three strings. Use configuration objects to instantiate custom trackers.

**Example**: creating a tracker with default configuration

Old API:

```java
BatchEmitter emitter = BatchEmitter.builder()

        .url("http://collectorEndpoint")

        .build();

Tracker tracker = new Tracker

        .TrackerBuilder(emitter, "trackerNamespace", "appId")

        .build();
```

New v1 API:

```java
Tracker tracker = Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");
```

**Example**: creating a tracker with custom configuration

Old API:

```java
BatchEmitter emitter = BatchEmitter.builder()

        .url("http://collectorEndpoint")

        .batchSize(10)

        .threadCount(1)

        .build();

Tracker tracker = new Tracker

        .TrackerBuilder(emitter, "trackerNamespace", "appId")

        .platform(DevicePlatform.Desktop)

        .base64(false)

        .build();
```

Version 1.0.0:

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("trackerNamespace", "appId")

        .platform(DevicePlatform.Desktop)

        .base64Encoded(false);

NetworkConfiguration networkConfig = new NetworkConfiguration("http://collectorEndpoint");

EmitterConfiguration emitterConfig = new EmitterConfiguration()

        .batchSize(10)

        .threadCount(1);

        

Tracker tracker = Snowplow.createTracker(trackerConfig, networkConfig, emitterConfig);
```

In the above examples, the `Tracker` is saved as a separate variable. That's not actually necessary with the new `Snowplow` class! Access stored `Tracker` objects by their namespace, or use the default tracker:

New v1 API:

```java
Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");

Snowplow.getDefaultTracker().track(event);
```

### `SelfDescribing` events

The `Unstructured` class has been renamed to `SelfDescribing`, bringing consistency with other Snowplow trackers. These are the custom events, based on self-describing (self-referential) JSONs, that allow you to create the perfect data for your business needs.

**Example**: replacing `Unstructured` with `SelfDescribing`

Old API:

```java
Unstructured event = Unstructured.builder()

            .eventData(dataAsSelfDescribingJson)

            .build();
```

New v1 API:

```java
SelfDescribing event = SelfDescribing.builder()

            .eventData(dataAsSelfDescribingJson)

            .build();
```

### Custom retry logic

The event sending retry logic has been refined. By default, the tracker will now not retry requests with the HTTP status codes 400, 401, 403, 410, or 422. This can be configured using the Emitter builder option `customRetryForStatusCodes()`, which replaces `fatalResponseCodes()`. Note that 2xx status codes will always be deemed successful, and will never be retried.

`customRetryForStatusCodes()` takes a map of integer codes and booleans - true for "yes, this should be retried" and false for "no, this should not be retried".

**Example**: setting custom retry rules

Old API:

```java
List<Integer> noRetry = new ArrayList<>();

noRetry.add(500);



BatchEmitter emitter = BatchEmitter.builder()

        .url("https://collector")

        .fatalResponseCodes(noRetry)

        .build();
```

New v1 API:

```java
Map<Integer, Boolean> customRetry = new HashMap<>();

customRetry.put(500, false);



BatchEmitter emitter = new BatchEmitter(

      new NetworkConfiguration("https://collector"),

      new EmitterConfiguration().customRetryForStatusCodes(customRetry));
```

### New method for `Emitter` interface

The `BatchEmitter` method `close()` has been added to the `Emitter` interface. This method stops the executor service and closes the threads, and may be required to cleanly exit your application. If you are using a custom `Emitter`, you'll need to implement the method (`@Override`). The `Tracker` class now has a `close()` method too, which calls `Emitter.close()`.

### Updated method for `EventStore` interface

The `EventStore` method `cleanupAfterSendingAttempt()` now returns `List<TrackerPayload>`, rather than `void`. This is to help with the new `EmitterCallback` implementation (see below). If you are using a custom `EventStore`, you'll need to update the method signature.

## Default configuration changes

The default `bufferCapacity`, the maximum number of events that can be buffered in memory, is now 10 000 (down from `Integer.MAX_VALUE`).

By default, the Java tracker now does not retry sending events if the request received the HTTP status code `400 Bad Request`, `401 Unauthorised`, `403 Forbidden`, `410 Gone`, or `422 Unprocessable Entity`.

When requests get a "unsuccessful" response status code - anything other than 2xx - a backoff wait time is added before the next request is processed. The wait time increases exponentially with every failure. There is now a maximum wait time of 10 minutes.

## Removed features

The `SimpleEmitter` class, deprecated in v0.12, has been removed. The default `Emitter` is the `BatchEmitter`, but you can also create your own `Emitter` using the provided interface. The `BatchEmitter` sends events in batches via POST. Deleting `SimpleEmitter` allowed us to tidy up behind-the-scenes, merging `AbstractEmitter` and `BatchEmitter` into one class.

We also removed the [Guava](https://github.com/google/guava) dependency.

## New features

New and improved callbacks! You can now add custom callbacks using the `EmitterCallback` interface, for example to create tracker metrics. The `EmitterCallback` will be called when events are successfully sent, and also in response to several different failures. It's described fully [here](/docs/sources/java-tracker/configuring-how-events-are-sent/#using-the-emitter-callback).

You can now add a `CookieJar` to persist the third-party event collector `sp` cookie across requests. This allows all events to have the same `network_userid`.

---

# Track events with the Java tracker
> Track page views, screen views, structured events, and custom self-describing events with the Java tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/tracking-events/

To track an event, pass an `Event` instance to the `Tracker`.

For example, tracking a ScreenView:

```java
Event event = new ScreenView.builder()

                .name("screen name")

                .build();

tracker.track(event);
```

The Java tracker makes it easy to track different kinds of data. We provide a range of `Event` classes for tracking out-of-the-box event types as well as fully custom events.

Every tracked event payload has a unique `event_id` UUID string. Other ubiquitous properties include the `name_tracker` (`trackerNamespace`) and `app_id` (`appId`) set when the `Tracker` was initialized. From version 0.12 onwards, `Tracker.track()` returns the payload's `eventId`.

Snowplow events have a defined structure and [protocol](/docs/events/) that is identical regardless of the tracker used. A minimal payload - the raw event - is sent from the tracker to your collector. The raw event is [enriched](/docs/pipeline/enrichments/) as it passes through your pipeline. By the time the event arrives in your data storage, depending which [enrichments](/docs/pipeline/enrichments/available-enrichments/) you have enabled, it will have gained different kinds of metadata, and have many more fields than it started with. The default Java tracker event fields are shown [here](/docs/sources/java-tracker/what-do-java-tracker-events-look-like/).

The [Java tracker Github repository](https://github.com/snowplow/snowplow-java-tracker) includes a mini demo, "simple-console". The demo sends one event of each type to your event collector.

## Auto-tracked events

The Java tracker does not yet support automatic event tracking. All tracking must be implemented manually.

## Manually-tracked events summary

The Java tracker provides classes for tracking different types of events. They are listed below.

| `Event` class              | `e` in raw event | `eventType` in enriched event |
| -------------------------- | ---------------- | ----------------------------- |
| `SelfDescribing` (custom)  | ue               | unstruct                      |
| `ScreenView`               | ue               | unstruct                      |
| `Timing`                   | ue               | unstruct                      |
| `PageView`                 | pv               | page\_view                    |
| `Structured`               | se               | struct                        |
| `EcommerceTransaction`     | tr               | transaction                   |
| `EcommerceTransactionItem` | ti               | transaction\_item             |

> **Note:** `EcommerceTransaction`/`EcommerceTransactionItem` are a legacy design and will be deprecated soon.

`SelfDescribing` events (called `Unstructured` prior to v1) allow you to track anything that can be described by a [JSON schema](/docs/fundamentals/schemas/). The data you provide will be sent as a JSON inside the raw event payload. The specific type of JSON schema needed are described fully on the [next page](/docs/sources/java-tracker/custom-tracking-using-schemas/).

The `ScreenView` and `Timing` out-of-the-box event types are actually wrappers for `SelfDescribing` events: the methods for building those events represent fields in their hidden self-describing JSON schemas. This is why both `ScreenView` and `Timing` events are labelled "unstruct" in the data warehouse.

The `PageView` and `Structured` event types are processed differently from `SelfDescribing` events. Properties tracked using these events are not sent packaged as JSON, but as individual "atomic" fields in the raw event payload. We call these event types "canonical" or "primitive".

`SelfDescribing` events and canonical events are loaded into and modelled differently in your data warehouse. The "atomic" fields will always have individual columns.

`EcommerceTransaction` and `EcommerceTransactionItem` events are legacy primitive events. We recommend instead designing your own `SelfDescribing` events plus [context entities](/docs/sources/java-tracker/custom-tracking-using-schemas/) for eCommerce tracking.

### Tracking data that is not event-type specific

Some data, such as that relating to the user whose activity is being tracked, is relevant across all event types. The Java tracker provides two mechanisms for tracking this kind of data.

Certain properties, including `userId` or `ipAddress`, can be set as "atomic" properties in the raw event, using the `Subject` class. `Subject` properties relate mainly to client-side tracking. If you are using the Java tracker for server-side tracking, you may wish to pass client-side data for tracking server-side. This is discussed [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/).

A more general and powerful method is to attach self-describing JSON "context entities" to your events - the same JSON schemas as used for `SelfDescribing` events. This means that any data that can be described by a JSON schema can be added to any or all of your events. Read more on the [next page](/docs/sources/java-tracker/custom-tracking-using-schemas/).

All events also provide the option for setting a custom timestamp, called `trueTimestamp`. See below for details.

### Creating a custom event (SelfDescribing)

To track data using an `SelfDescribing` event, the data must be structured as a `SelfDescribingJson` object, discussed fully on the next page. These require two fields. The first is a URI for a self-describing JSON schema. The second is a data map, and the data must be valid against the schema. `SelfDescribing` events can be considered wrappers for sending `SelfDescribingJson`.

The simplest initialisation looks like this:

```java
SelfDescribing selfDescribing = SelfDescribing.builder()

            .eventData(dataAsSelfDescribingJson)

            .build();
```

See the API docs for the full [SelfDescribing.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/SelfDescribing.Builder.html) options.

### Creating a `ScreenView` event

Track screen views with the `ScreenView` event. It's a wrapper around a `SelfDescribing` event, using [this schema](https://github.com/snowplow/iglu-central/blob/324cfe9a5390174a56084722a545287d01a5a060/schemas/com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0).

A simple initialisation looks like this:

```java
ScreenView screenView = ScreenView.builder()

            .name("human readable screen name")

            .id("unique screen ID")

            .build();
```

At least one of `name` or `id` must be set. See the API docs for the full [ScreenView.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/ScreenView.Builder.html) options.

### Creating a `Timing` event

Track how long something took with the `Timing` event. It's a wrapper around a `SelfDescribing` event, using [this schema](https://github.com/snowplow/iglu-central/blob/324cfe9a5390174a56084722a545287d01a5a060/schemas/com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0).

A simple initialisation looks like this:

```java
Timing timing = Timing.builder()

            .category("category of the timed event")

            .variable("name of the timed event")

            .timing(10) // in milliseconds

            .label("optional label")

            .build();
```

Provide your `timing` value in milliseconds. The `label` property is optional. See the API docs for the full [Timing.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/Timing.Builder.html) options.

### Creating a PageView event

Track page views with the `PageView` event. This is a "canonical" event type; data will end up in individual "atomic" columns in the data warehouse.

| Property     | Field in raw event | Column in enriched event |
| ------------ | ------------------ | ------------------------ |
| page URL     | url                | page\_url                |
| page title   | page               | page\_title              |
| referrer URL | refr               | page\_referrer           |

The provided URLs will also be decomposed into other columns, such as `page_urlscheme`, during event [enrichment](/docs/pipeline/enrichments/).

A simple initialisation looks like this:

```java
PageView pageViewEvent = PageView.builder()

            .pageUrl("https://www.snowplowanalytics.com")

            .pagetitle("Snowplow")

            .referrer("https://github.com/snowplow/snowplow-java-tracker")

            .build();
```

Only `pageUrl` is required. See the API docs for the full [PageView.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/PageView.Builder.html) options.

### Creating a `Structured` event

To track custom data without schemas, use `Structured` events. They are the "canonical" equivalent of `SelfDescribing` events. The provided data will end up in individual "atomic" columns in the data warehouse. Because of this, it's not possible to fully customise a `Structured` event: the fields cannot be renamed, nor new fields added. `Structured` events are designed to be similar to Google-style events.

The `Structured` event fields have flexible definitions, and what you put into each field is up to you. This is a double-edged sword. It's highly advisable to agree business-wide on definitions for each of these fields, before implementing tracking.

| Property | Often contains data about     | Field in raw event | Column in enriched event |
| -------- | ----------------------------- | ------------------ | ------------------------ |
| category | Grouping for the action       | se\_ca             | se\_category             |
| action   | Type of user activity         | se\_ac             | se\_action               |
| label    | Additional event data         | se\_la             | se\_label                |
| property | The action or object acted on | se\_pr             | se\_property             |
| value    | Numerical event data          | se\_va             | se\_value                |

A simple initialisation looks like this:

```java
Structured structured = Structured.builder()

                .category("category e.g. auth")

                .action("action e.g. logout")

                .label("optional label")

                .property("optional property")

                .value(12.34)

                .build();
```

Both `category` and `action` are required. See the API docs for the full [Structured.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/Structured.Builder.html) options.

### Creating `EcommerceTransaction` and `EcommerceTransactionItem` events

To track eCommerce data, we recommend designing your own schemas for an `SelfDescribing` event. We suggest creating one schema for the overall transaction, and another schema for individual items. The transaction schema can be used for the `SelfDescribing` event. Items in the transaction can be added as [context entities](/docs/sources/java-tracker/custom-tracking-using-schemas/).

The `EcommerceTransaction` and `EcommerceTransactionItem` events are legacy events, and it's highly likely they will be deprecated in the future. They are designed to be sent together, with one `EcommerceTransaction` containing an `EcommerceTransactionItem` for every item in the transaction. When the `EcommerceTransaction` event is tracked, the `EcommerceTransactionItem` events are extracted and sent separately. This means that although you have only tracked one event (using `Tracker.track()`), multple events are generated.

`EcommerceTransaction` and `EcommerceTransactionItem` are "canonical" events. The data will end up in individual "atomic" columns in the data warehouse.

| `EcommerceTransaction` property | Field in raw event | Column in enriched event |
| ------------------------------- | ------------------ | ------------------------ |
| orderId                         | tr\_id             | tr\_orderid              |
| totalValue                      | tr\_tt             | tr\_total                |
| affiliation                     | tr\_af             | tr\_affiliation          |
| taxValue                        | tr\_tx             | tr\_tax                  |
| shipping                        | tr\_sh             | tr\_shipping             |
| city                            | tr\_ci             | tr\_city                 |
| state                           | tr\_st             | tr\_state                |
| country                         | tr\_co             | tr\_country              |
| currency                        | tr\_cu             | tr\_currency             |

| `EcommerceTransactionItem` property | Field in raw event | Column in enriched event |
| ----------------------------------- | ------------------ | ------------------------ |
| itemId                              | ti\_id             | ti\_orderid              |
| sku                                 | ti\_sk             | ti\_sku                  |
| price                               | ti\_pr             | ti\_price                |
| quantity                            | ti\_qu             | ti\_quantity             |
| name                                | ti\_nm             | ti\_name                 |
| category                            | ti\_ca             | ti\_category             |
| currency                            | ti\_cu             | ti\_currency             |

The `orderId` and `itemId` should be the same, as it's the most direct way to associate the two events once they are in the warehouse.

A simple initialisation looks like this:

```java
// Create events for each item in the transaction

EcommerceTransactionItem item = EcommerceTransactionItem.builder()

        .itemId("should be the same as order_id")

        .sku("sku")

        .price(1.0)

        .quantity(2)

        .name("item name")

        .category("category")

        .currency("currency")

        .build();



// Then make the EcommerceTransaction event

EcommerceTransaction ecommerceTransaction = EcommerceTransaction.builder()

        .items(item) // Add the EcommerceTransactionItem events

        .orderId("should be the same as item_id")

        .totalValue(2.0)

        .affiliation("affiliation")

        .taxValue(2.0)

        .shipping(3.0)

        .city("city")

        .state("state")

        .country("country")

        .currency("currency")

        .build();
```

For `EcommerceTransactionItem` events, `itemId`, `sku`, `price` and `quantity` are required. For `EcommerceTransaction` events, only `orderId` and `totalValue` are required. See the API docs for the full [EcommerceTransaction.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/EcommerceTransaction.Builder.html) and [EcommerceTransactionItem.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/EcommerceTransactionItem.Builder.html) options.

### Adding custom timestamps to events

Snowplow events have several timestamps. The raw event payload always contains a `deviceCreatedTimestamp` (`dtm`) and a `deviceSentTimestamp` (`stm`). Other timestamps are added as the event moves through the pipeline.

Every `Event.Builder` in the Java tracker allows for a custom timestamp, called `trueTimestamp` to be set. A `trueTimestamp` can be added to any event using the `trueTimestamp()` Builder method:

```java
// This example shows an Unstructured event, but all events can have a trueTimestamp

SelfDescribing selfDescribing = SelfDescribing.builder()

            .eventData(dataAsSelfDescribingJson)

            .trueTimestamp(1647616394785L)

            .build();
```

`trueTimestamp` should be a `Long` representing milliseconds since the Unix epoch.

---

# Track client side properties with the Java tracker
> Add user properties like user ID, IP address, and device information to events using the Subject class for client-side tracking.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/tracking-specific-client-side-properties/

An event describes a single, transient activity. The context in which that event occurs - the relatively persistent environment - is also incredibly valuable data. As discussed on the previous page, the most powerful way to track such data is to use [custom self-describing JSON entities](/docs/sources/java-tracker/custom-tracking-using-schemas/). These are sent as JSON in the raw event payload. Depending on your data warehouse, event context entities are loaded into their own tables, or remain as JSON in the loaded event.

The Java tracker provides an additional "canonical" method to track a small subset of contextual data, using the `Subject` class. As with the "canonical" [Event types](/docs/sources/java-tracker/tracking-events/), e.g. `PageView`, `Subject` properties directly populate the "atomic" fields of the raw event payload. These data will always end up in individual columns of the main event table in the data warehouse.

The fields tracked using `Subject` tend to be most relevant in client-side tracking. Some are set automatically in all events during enrichment, even when no `Subject` is added. These properties are marked with `*` below, and discussed below. Timezone, marked with `**`, is set when a `Subject` is tracked with the event.

Add these fields to an event using `Subject`:

| Property         | Field in raw event | Column(s) in enriched event           |
| ---------------- | ------------------ | ------------------------------------- |
| userId           | uid                | user\_id                              |
| ipAddress\*      | ip                 | user\_ipaddress                       |
| timezone\*\*     | tz                 | os\_timezone                          |
| language         | lang               | br\_lang                              |
| useragent\*      | ua                 | useragent                             |
| viewport         | vp                 | br\_viewheight, br\_viewwidth         |
| screenResolution | res                | dvce\_screenheight, dvce\_screenwidth |
| colorDepth       | cd                 | br\_colordepth                        |
| networkUserId\*  | tnuid              | network\_userid                       |
| domainUserId     | duid               | domain\_userid                        |
| domainSessionId  | sid                | domain\_sessionid                     |

Note: the ability to set `domainSessionId` was added in version 0.11.

These properties are especially useful for client-side tracking, or for linking client-side and server-side tracking. We highly recommend tracking in both client-side and server-side, if it's relevant to your application.

If you are also using the Javascript tracker, it will set [cookies](/docs/sources/web-trackers/cookies-and-local-storage/) in the browser. The `Subject` properties `domainUserId`, and `domainSessionId` are intended to be used for extracted cookie values passed to the Java tracker. If you want to track other identification tokens, we recommend creating schemas and using context entities.

As always, be aware of privacy when tracking personal identifiable information such as email addresses or IP addresses.

### Overriding autogenerated event properties

All enriched Snowplow events contain values for `user_ipaddress`, `useragent`, and `network_userid`.

The `user_ipaddress` is automatically added to all enriched events. To manually override this, use a `Subject` and set a `ipAddress` string; use an empty string to prevent IP address tracking. Alternatively, use the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/).

The `useragent` is also automatically added during enrichment. Snowplow pipelines provide multiple useragent-parsing [enrichments](/docs/pipeline/enrichments/available-enrichments/). To manually override the detected useragent, use a `Subject` and set a `useragent` string. For the default Tracker configuration with OkHttp, the default `useragent` will be "okhttp/4.2.2".

The `network_userid` is the cookie value for the event collector’s third-party cookie. It is the server-side user identifier. The cookie is named `sp` (or `micro` for Snowplow Micro pipelines). The default behavior is for the collector to provide a new cookie/`network_userid` for each event it receives. To override the collector cookie’s value with your own generated ID, use a `Subject` object and set `networkUserId`. Alternatively, configure a `CookieJar` to persist the `sp` cookie, and the `network_userid`, across requests (see [here](/docs/sources/java-tracker/configuring-how-events-are-sent/#persisting-cookies-using-a-cookiejar)).

A further property, `timezone`, is generated automatically during `Subject` initialization, based on `Calendar.getInstance().getTimeZone()`. Therefore, this will be added to all events with a `Subject` attached. The default will be overriden if `timezone` is provided explicitly.

### Adding `Subject` properties to events

There are two ways to track the `Subject` "atomic" properties in your events: on an event-by-event basis; or globally, to affect all events. A combination of both methods is also possible.

A simple `Subject` initialisation looks like this:

```java
SubjectConfiguration subjectConfig = new SubjectConfiguration()

        .userId("java@snowplowanalytics.com");

Subject subject = new Subject(subjectConfig);
```

In this case only the `userId` (and `timezone`) were set. There are no required properties for the `SubjectConfiguration`.

A `Subject` can be added to any event using the `subject()` `Event.Builder` method:

```java
// This example shows an SelfDescribing event, but all events can have a Subject

SelfDescribing selfDescribing = SelfDescribing.builder()

            .eventData(dataAsSelfDescribingJson)

            .subject(subject)

            .build();
```

To set `Subject` properties in all subsequent events, add a `Subject` to your `Tracker` object:

```java
// A Subject can be provided at Tracker initialisation

// Indirectly, using the Snowplow interface and a SubjectConfiguration object

SubjectConfiguration subjectConfig = new SubjectConfiguration().useragent("useragent");

Tracker tracker = Snowplow.createTracker(

        new TrackerConfiguration("trackerNamespace", "appId"),

        new NetworkConfiguration("http://collector-url"),

        subjectConfig);



// Creating a Tracker directly using Subject and Emitter objects

Subject subject = new Subject();

BatchEmitter emitter = new BatchEmitter(new NetworkConfiguration("http://endpoint"));

Tracker tracker = new Tracker(new TrackerConfiguration("trackerNamespace", "appId"), emitter, subject);



// Alternatively, a Subject can be set later

Tracker tracker = Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");

Subject subject = new Subject();

tracker.setSubject(subject);
```

Subject properties can be updated or added to after initialization, using setter methods. See the [API docs](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Subject.html) for full details.

It's possible to use both `Event`-specific and `Tracker`-associated `Subject` objects simultaneously. Fields from both `Subject` objects are added to the payload, with the `Event`-specific `Subject` having priority.

```java
// Adding a global, Tracker-associated Subject

SubjectConfiguration trackerSubjectConfig = new SubjectConfiguration()

        .language("EN")

        .useragent("Snowplow");



Tracker tracker = Snowplow.createTracker(

        new TrackerConfiguration("trackerNamespace", "appId"),

        new NetworkConfiguration("http://collector-url"),

        trackerSubjectConfig);



// Adding an Event-specific Subject

SubjectConfiguration eventSubjectConfig = new SubjectConfiguration()

        .userId("java@snowplowanalytics.com")

        .useragent("Mozilla/5.0");



SelfDescribing selfDescribing = SelfDescribing.builder()

        .eventData(dataAsSelfDescribingJson)

        .subject(new Subject(eventSubjectConfig))

        .build();



// Tracking the event

tracker.track(selfDescribing);
```

The resulting enriched event would have these `Subject` atomic columns populated:

| Column in enriched event | Value / example value                       | Source         |
| ------------------------ | ------------------------------------------- | -------------- |
| user\_id                 | "<java@snowplowanalytics.com>"              | eventSubject   |
| os\_timezone             | e.g. "Europe/London"                        | eventSubject   |
| br\_lang                 | "EN"                                        | trackerSubject |
| useragent                | "Mozilla/5.0"                               | eventSubject   |
| network\_userid          | e.g. "8383057f-2769-4321-ad72-58fa1b22e4b3" | pipeline       |

---

# Using multiple trackers in one Java app
> Instantiate and manage multiple Java tracker instances with different configurations using the Snowplow interface.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/using-multiple-trackers/

Rarely, a tracking implementation may benefit from instantiating multiple Java trackers, each with a different configuration. This will not be appropriate for most Snowplow users.

Creating and managing multiple trackers is easiest using the `Snowplow` interface (added in v1.0.0). All trackers created with `Snowplow.createTracker()` are added to a private static map. Different `Tracker` objects are distinguished using their namespaces.

In this example, the user wants to send some events to a different collector. They create two Trackers, one for each endpoint:

```java
Snowplow.createTracker("main_tracker", "appId", "http://endpoint");

Snowplow.createTracker("other_tracker", "appId", "http://other-endpoint");



Snowplow.getDefaultTracker().track(event);

Snowplow.getTracker("other_tracker").track(event2);
```

See the API docs for the full [Snowplow](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Snowplow.html) details.

## Using different Snowplow trackers together

Because all Snowplow trackers create events with the same structure, it's easy to work with data from different Snowplow tracker SDKs. For example, you may wish to implement the Java tracker for back-end server tracking, in conjunction with a client-side tracker such as JavaScript or Flutter.

Read more about tracking client-side/server-side [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/).

---

# What do Java tracker events look like?
> Understand the structure of enriched Java tracker events with example JSON payloads for different event types.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/what-do-java-tracker-events-look-like/

See [Tracking events](/docs/sources/java-tracker/tracking-events/) to learn how to track events with the Java tracker.

All Snowplow events, regardless of the tracker used, look similar. They all currently contain 137 columns/properties. Some are populated by pipeline enrichments, e.g. the [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) populates the `mkt_` columns. Snowplow event structure and columns are explained [here](/docs/fundamentals/canonical-event/).

This reference page shows two simple example events generated using the default tracker configuration and a [Snowplow Micro](/docs/testing/snowplow-micro/) testing pipeline. These events were tracked from a simplified version of the "simple-console" demo provided in the [Java tracker Github repository](https://github.com/snowplow/snowplow-java-tracker). While most Snowplow pipelines transform events into table rows, Micro has an API that provides events in JSON format.

The JSON below is part of the "event" value for a [ScreenView](/docs/sources/java-tracker/tracking-events/#creating-a-screenview-event) event, which is a type of [SelfDescribing](/docs/sources/java-tracker/tracking-events/#creating-a-custom-event-selfdescribing) event. The full Micro JSON output includes other data, such as headers. Any columns with `null` values have also been removed.

```java
// This event was tracked

ScreenView screenView = ScreenView.builder()

            .name("login")

            .id("auth123")

            .build();
```

```json
"ScreenView event (edited)": {

  "app_id": "java-tracker-sample-console-app",

  "platform": "srv",

  "etl_tstamp": "2022-03-22T15:49:56.999Z",

  "collector_tstamp": "2022-03-22T15:49:56.957Z",

  "dvce_created_tstamp": "2022-03-22T15:49:56.734Z",

  "event": "unstruct",

  "event_id": "f9a58d5d-cd13-4bfe-b9b2-5136ce1cb9fa",

  "name_tracker": "demo",

  "v_tracker": "java-0.12.0",

  "v_collector": "ssc-2.3.0-stdout$",

  "v_etl": "snowplow-micro-1.1.2-common-2.0.1",

  "user_ipaddress": "manually removed for privacy",

  "network_userid": "666219d1-9b9c-4fe1-b694-c8e94dd5a706",

  "unstruct_event": {

    "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",

    "data": {

      "schema": "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0",

      "data": { "id": "auth123", "name": "login" }

    }

  },

  "useragent": "okhttp/4.2.2",

  "dvce_sent_tstamp": "2022-03-22T15:49:56.757Z",

  "derived_tstamp": "2022-03-22T15:49:56.934Z",

  "event_vendor": "com.snowplowanalytics.snowplow",

  "event_name": "screen_view",

  "event_format": "jsonschema",

  "event_version": "1-0-0",

}
```

Most of these properties are the default Java tracker properties, which will be present in all Java tracker events. For example, `app_id`,`platform`, and `name_tracker` were set at `Tracker` initialization.

Since this is a `SelfDescribing` event (previously called `Unstructured`), the "unstruct\_event" field is also populated. The [self-describing JSON](/docs/sources/java-tracker/custom-tracking-using-schemas/) provided as part of the `SelfDescribing`/`ScreenView` event has been validated against the [Iglu](/docs/api-reference/iglu/) schema registry.

Below is an example `PageView` event using the default tracker configuration with Snowplow Micro. Again, the `null` columns and other sections of the payload have been removed.

```java
// This event was tracked

PageView pageViewEvent = PageView.builder()

            .pageUrl("https://www.snowplowanalytics.com")

            .build();
```

```json
"PageView event (edited)": {

  "app_id": "java-tracker-sample-console-app",

  "platform": "srv",

  "etl_tstamp": "2022-03-22T17:02:44.394Z",

  "collector_tstamp": "2022-03-22T17:02:44.268Z",

  "dvce_created_tstamp": "2022-03-22T17:02:43.913Z",

  "event": "page_view",

  "event_id": "64ae291c-2170-4643-a957-0b2f766c8ac8",

  "name_tracker": "demo",

  "v_tracker": "java-0.12.0",

  "v_collector": "ssc-2.3.0-stdout$",

  "v_etl": "snowplow-micro-1.1.2-common-2.0.1",

  "user_ipaddress": "manually removed for privacy",

  "network_userid": "bbf5eda0-f3f7-48b9-9068-b12e721a7b9c",

  "page_url": "https://www.snowplowanalytics.com",

  "page_urlscheme": "https",

  "page_urlhost": "www.snowplowanalytics.com",

  "page_urlport": 443,

  "useragent": "okhttp/4.2.2",

  "dvce_sent_tstamp": "2022-03-22T17:02:43.926Z",

  "derived_tstamp": "2022-03-22T17:02:44.255Z",

  "event_vendor": "com.snowplowanalytics.snowplow",

  "event_name": "page_view",

  "event_format": "jsonschema",

  "event_version": "1-0-0",

}
```

Because PageView is a "primitive" [event type](/docs/sources/java-tracker/tracking-events/), the provided URL has populated the "atomic" `page_url` column. The different parts of the URL - scheme and host - have been automatically extracted and added to their own columns.

---

# Lua tracker SDK
> Track events in Lua applications with the Snowplow Lua tracker SDK, compatible with Lua 5.1+ and LuaJIT.
> Source: https://docs.snowplow.io/docs/sources/lua-tracker/

The Snowplow Lua Tracker allows you to track Snowplow events in your Lua applications.

The current version is 0.2.0. It is compatible with Lua >= 5.1, along with LuaJIT.

This will help you set up the Lua tracker and get started tracking. For more technical details, you can visit the [API docs](https://snowplow.github.io/snowplow-lua-tracker/).

## Getting Started

### Requirements

Ensure you have the following installed on your system:

- [Lua](https://www.lua.org/) version >= 5.1
- [LuaRocks](https://luarocks.org/) (Lua's dependency manager)
- curl

If using `brew`, simply run:

```bash
brew install lua luarocks curl
```

### Installation

Once these are installed, you can install the tracker using LuaRocks:

```bash
luarocks install snowplowtracker
```

Note: You may find that you need to pass in the CURL\_DIR flag if Lua cannot find `curl` by itself. Below is an example if `curl` was installed using `brew` on an Intel Mac.

```bash
luarocks install snowplowtracker CURL_DIR=/usr/local/Cellar/curl/7.82.0/
```

Or, add the Snowplow Tracker to the dependencies section of your rockspec (Note: you may still have to pass `CURL_DIR` when installing):

```gradle
dependencies = {

  ...

  "SnowplowTracker"

}
```

### Tracking Events

To track an event, simply create a tracker instance and call one of the `track_*` methods. For example, simple tracking of a [structured event](/docs/events/custom-events/#structured-events) can be done as follows:

```lua
local snowplow = require("snowplow")

local tracker = snowplow.new_tracker("{{ collector_url }}")



tracker:track_struct_event("category", "action", "label", "property", 10)
```

Visit documentation about [tracking events](/docs/sources/lua-tracker/tracking-specific-events/) to learn about other supported event types.

---

# Track events with the Lua tracker
> Track screen views, structured events, and custom self-describing events with the Lua tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/lua-tracker/tracking-specific-events/

Creating a tracking instance is as simple as calling `snowplow.new_tracker` and providing a URL for your collector:

```lua
local snowplow = require("snowplow")

local tracker = snowplow.new_tracker("{{ collector_url }}")
```

Tracking methods supported by the Lua Tracker:

| Method                         | Event type tracked                                  |
| ------------------------------ | --------------------------------------------------- |
| track\_screen\_view            | View of screen                                      |
| track\_struct\_event           | Semi-custom structured event                        |
| track\_self\_describing\_event | Custom event based on “self-describing” JSON schema |

## Track structured events with `track_struct_event`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. “Struct” events closely mirror the structure of Google Analytics events, with “category”, “action”, “label”, and “value” properties.

| Parameter | Description                                                    | Required in event? |
| --------- | -------------------------------------------------------------- | ------------------ |
| category  | The grouping of structured events which this action belongs to | Yes                |
| action    | Defines the type of user interaction which this event involves | Yes                |
| label     | Often used to refer to the ‘object’ the action is performed on | No                 |
| property  | Describing the ‘object’, or the action performed on it         | No                 |
| value     | Provides numerical data about the event                        | No                 |

```lua
tracker:track_struct_event("shop", "add-to-basket", "book", "pcs", 2)
```

## Track self-describing events with `track_self_describing_event`

Use `track_self_describing_event` to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure. A guide to understanding Self-Describing events is [available here](/docs/fundamentals/events/#self-describing-events).

| Parameter | Description                            | Required in event? |
| --------- | -------------------------------------- | ------------------ |
| schema    | The schema to use                      | Yes                |
| action    | The data to send along with the schema | Yes                |

```lua
tracker:track_self_describing_event(

  "iglu:com.snowplowanalytics.snowplow/add_to_cart/jsonschema/1-0-0",

  { sku = "ASO01043", unitPrice = 49.95, quantity = 1000 }

)
```

## Track screen views with `track_screen_view`

Use track\_screen\_view to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages.

This method uses a self-describing event with the [`com.snowplowanalytics.snowplow/screen_view` schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0).

```lua
tracker:track_screen_view("Character Creation - Step 1", "c1")
```

## Configuring the Tracker

The `new_tracker` method has a couple of other parameters that can be used to configure your tracker instance, along with the collector `url`.

| Parameter      | Description                                          | Required | Default |
| -------------- | ---------------------------------------------------- | -------- | ------- |
| url            | The Snowplow Collector URL                           | Yes      | -       |
| request\_type  | The request type to use ("GET" or "POST")            | No       | "POST"  |
| encode\_base64 | If self-describing event payloads are base64 encoded | No       | true    |

Note: It's generally recommended to use the default values as `encode_base64 = true` can reduce the size of payloads and `POST` will receive more future support (such as batched requests), but you can change these if your use case calls for it.

## Tracker Method Return Values

A call to a `track_*` method will return two values, a boolean if the request was successful, and an optional error message if the request was not successful.

An example of the values returned from a failed request:

```lua
local ok, err = tracker:track_screen_view("Character Configuration - Part 1", "c1")

-- false, Host [https://test.invalid/com.snowplowanalytics.snowplow/tp2] not found (possible connectivity error)
```

An example of the values returned from a successful request:

```lua
local ok, err = tracker:track_screen_view("Character Configuration - Part 1", "c1")

-- true, nil
```

## Adding user and platform data

The tracker can store information about the user associated with the event, such as their `user_id`, what type of device they used, or what size screen that device had. It also stores which platform the event occurred on – e.g. server-side app, mobile, games console, etc. This is done through the provided `set_*` methods available on a tracker instance. The stored information is attached to the tracked events using fields described in the [Tracker Protocol](/docs/events/).

For a full list of setters, check out the [Tracker API Documentation](https://snowplow.github.io/snowplow-lua-tracker/modules/Tracker.html).

---

# Google Play Data safety (Android)
> Guidance for declaring data collection and handling practices when publishing Android apps using the Snowplow tracker to Google Play.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/android-google-play-data-safety/

Google Play require developers to declare how they collect and handle user data within their app. The provided information is shown on the Google Play store listing.

This table provides guidance for publishing apps incorporating our Snowplow Android Tracker.

| Category                    | Data type                    | Description                                                                                                                                                                                     | Snowplow Android tracker feature                                                                                                                                                                                                        |
| --------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Location                    | Approximate location         | User or device physical location to an area greater than or equal to 3 square kilometers, such as the city a user is in, or location provided by Android's `ACCESS_COARSE_LOCATION` permission. | Geolocation context (optional and disabled by default in `TrackerConfiguration` `geoLocationContext`)                                                                                                                                   |
|                             | Precise location             | User or device physical location within an area less than 3 square kilometers, such as location provided by Android's `ACCESS_FINE_LOCATION` permission.                                        |                                                                                                                                                                                                                                         |
| App activity                | Page views and taps in app   | Information about how a user interacts with your app. For example, the number of page views or the screen coordinates of taps.                                                                  | ScreenView automatic tracking (optional but enabled by default in `TrackerConfiguration` `screenViewAutotracking`)                                                                                                                      |
|                             | In-app search history        | Information about what a user has searched for in your app.                                                                                                                                     | Not tracked                                                                                                                                                                                                                             |
|                             | Installed apps               | Information about the apps installed on a user's device.                                                                                                                                        | Not tracked                                                                                                                                                                                                                             |
|                             | Other user-generated content | Any other user-generated content not listed here, or in any other section. For example, user bios or notes.                                                                                     | Not tracked                                                                                                                                                                                                                             |
|                             | Other actions                | Any other user activity or actions in-app not listed here such as gameplay and likes.                                                                                                           | Session automatic tracking (optional but enabled by default in `TrackerConfiguration` `sessionContext`). Background–Foreground transition tracking (optional and disabled by default in `TrackerConfiguration` `lifecycleAutotracking`) |
| Web browsing                | Web browsing history         | Information about the websites a user has visited.                                                                                                                                              | Not tracked                                                                                                                                                                                                                             |
| App info and performance    | Crash logs                   | Crash log data from your app. For example, the number of times your app has crashed, stack traces, or other information directly related to a crash.                                            | Crash reporting automatic tracking (optional but enabled by default in `TrackerConfiguration` `exceptionAutotracking`)                                                                                                                  |
|                             | Diagnostics                  | Information about the performance of your app. For example battery life, loading time, latency, framerate, or any technical diagnostics.                                                        | Platform context (optional but enabled by default in `TrackerConfiguration` `platformContext`)                                                                                                                                          |
|                             | Other app performance data   | Any other app performance data not listed here.                                                                                                                                                 | Not tracked                                                                                                                                                                                                                             |
| Device or other identifiers | Device or other identifiers  | Identifiers that relate to an individual device, browser or app. For example, an IMEI number, MAC address, Widevine Device ID, Firebase installation ID, or advertising identifier.             | Advertising identifier (AAID, also called IDFA) (optional, enabled by default as part of `TrackerConfiguration` `platformContext` but also requires extra configuration)                                                                |

> **Note:** The IP address of the device can be tracked by the collector. Disable this using [server-side anonymisation](/docs/sources/mobile-trackers/anonymous-tracking/).

---

# Anonymous tracking in the native mobile trackers
> Enable anonymous tracking to mask user identifiers and session information in mobile tracker events for privacy compliance.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/anonymous-tracking/

> **Info:** This feature is available since version 4.

Anonymous tracking is a feature that enables anonymization of various user and session identifiers, to support user privacy when consent for tracking the identifiers isn't given.

On mobile, the following [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) can be anonymized:

- Client-side user identifiers:

  - `user_id`, `domain_userid`, `network_userid`, and `user_ipaddress`, set by you in `Subject`
  - `userId`, a device ID, set automatically in the [session](/docs/sources/web-trackers/tracking-events/session/) entity
  - IDFA identifiers (`appleIdfa` and `appleIdfv` for iOS, `androidIdfa` for Android) in the mobile entity

- Client-side session identifiers:
  - `previousSessionId` in the session entity, set automatically by the tracker

- Server-side user identifiers:
  - `network_userid` and `user_ipaddress`, set by the [Collector](/docs/pipeline/collector/)

> **Note:** The Collector captures the IP address from the request HTTP headers, and updates the `user_ipaddress` event property. However, if you set the `user_ipaddress` property in the `Subject`, that value has priority.
>
> Similarly, if you set the `network_userid` property in the `Subject`, that value is used instead of the Collector cookie value.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

There are several levels to the anonymization depending on which of the categories are affected.

## 1. Full client-side anonymization

In this case, we want to anonymize both the client-side user identifiers as well as the client-side session identifiers. This means disabling the Session entity altogether and enabling user anonymization:

**iOS:**

```swift
let config = TrackerConfiguration()

    .sessionContext(false) // Session context entity won't be added to events

    .userAnonymisation(true) // User identifiers in Platform context (IDFA and IDFV) will be anonymized
```

**Android (Kotlin):**

```kotlin
val config = TrackerConfiguration("appId")

    .sessionContext(false) // Session context entity won't be added to events

    .userAnonymisation(true) // User identifiers in Platform context (IDFA and IDFV) will be anonymized
```

**Android (Java):**

```java
TrackerConfiguration config = new TrackerConfiguration("appId")

    .sessionContext(false) // Session context entity won't be added to events

    .userAnonymisation(true); // User identifiers in Platform context (IDFA and IDFV) will be anonymized
```

***

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ no session entity                           |
| `sessionId`         | Session entity           | ❌ no session entity                           |
| `previousSessionId` | Session entity           | ❌ no session entity                           |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

## 2. Client-side anonymisation with session tracking

This setting disables client-side user identifiers are but tracks session information. In practice, this means that events track the Session entity but the `userId` property is a null UUID (`00000000-0000-0000-0000-000000000000`). If the Mobile (platform) entity is enabled, the IDFA identifiers will not be present.

**iOS:**

```swift
let config = TrackerConfiguration()

    .sessionContext(true) // Session entity is tracked with the session ID

    .userAnonymisation(true) // User identifiers in Session and Mobile (platform) entity are anonymized
```

**Android (Kotlin):**

```kotlin
val config: TrackerConfiguration = TrackerConfiguration("appId")

    .sessionContext(true) // Session entity is tracked with the session ID

    .userAnonymisation(true) // User identifiers in Session and Mobile (platform) entity are anonymized
```

**Android (Java):**

```java
TrackerConfiguration config = new TrackerConfiguration("appId")

    .sessionContext(true) // Session entity is tracked with the session ID

    .userAnonymisation(true); // User identifiers in Session and Mobile (platform) entity are anonymized
```

***

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ null UUID                                   |
| `sessionId`         | Session entity           | ✅                                             |
| `previousSessionId` | Session entity           | ❌                                             |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

> **Note:** When anonymous tracking is enabled or disabled using `tracker.setUserAnonymisation(true | false)`, the tracker starts a new session. This results in a new `sessionId`.

## 3. Server-side anonymization

Server-side anonymization affects user identifiers set server-side. In particular, these are the `network_userid` property set in server-side cookie and the user IP address. You can anonymize the properties using the `serverAnonymisation` flag in `EmitterConfiguration`:

**iOS:**

```swift
let config = EmitterConfiguration()

    .serverAnonymisation(true)
```

**Android (Kotlin):**

```kotlin
val config = EmitterConfiguration()

    .serverAnonymisation(true)
```

**Android (Java):**

```java
EmitterConfiguration config = new EmitterConfiguration()

    .serverAnonymisation(true);
```

***

| Identifier          | Location in event        | Included in event?                                  |
| ------------------- | ------------------------ | --------------------------------------------------- |
| `user_id`           | Atomic                   | ✅                                                   |
| `domain_userid`     | Atomic                   | ✅                                                   |
| `userId`            | Session entity           | ✅                                                   |
| `sessionId`         | Session entity           | ✅                                                   |
| `previousSessionId` | Session entity           | ✅                                                   |
| `appleIdfa`         | Mobile (platform) entity | ✅                                                   |
| `appleIdfv`         | Mobile (platform) entity | ✅                                                   |
| `androidIdfa`       | Mobile (platform) entity | ✅                                                   |
| `network_userid`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |

Setting the flag will add a `SP-Anonymous` HTTP header to requests sent to the Snowplow Collector. The Snowplow pipeline will take care of anonymizing the identifiers.

## 4. Full anonymization

Full anonymization combines client-side and server-side anonymization, to remove all user and session identifiers from events. This approach provides the highest level of privacy.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .sessionContext(false) // Session entity won't be added to events

    .userAnonymisation(true) // User identifiers in Platform context will be anonymized



let emitterConfig = EmitterConfiguration()

    .serverAnonymisation(true) // Collector won't track network_userid or user IP
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .sessionContext(false) // Session entity won't be added to events

    .userAnonymisation(true) // User identifiers in Platform context will be anonymized



val emitterConfig = EmitterConfiguration()

    .serverAnonymisation(true) // Collector won't track network_userid or user IP
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .sessionContext(false) // Session entity won't be added to events

    .userAnonymisation(true); // User identifiers in Platform context will be anonymized



EmitterConfiguration emitterConfig = new EmitterConfiguration()

    .serverAnonymisation(true); // Collector won't track network_userid or user IP
```

***

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌                   |
| `appleIdfv`         | Mobile (platform) entity | ❌                   |
| `androidIdfa`       | Mobile (platform) entity | ❌                   |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

---

# Track client side properties in the native mobile trackers
> Configure user and device properties to automatically attach to all mobile tracker events using SubjectConfiguration.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/client-side-properties/

An event describes a single, transient activity. The context in which that event occurs - the relatively persistent environment - is also incredibly valuable data.

The tracker allows the addition of a persistent set of information through the `SubjectConfiguration`, which represents the basic information about the user and the app. This data is added to every event as part of the [canonical event properties](/docs/fundamentals/canonical-event/).

| Property           | Description                 | Automatically added?  | Column(s) in enriched event             |
| ------------------ | --------------------------- | --------------------- | --------------------------------------- |
| `userId`           | User identifier.            | ❌                     | `user_id`                               |
| `ipAddress`        | User IP address.            | ✅                     | `user_ipaddress`                        |
| `timezone`         | Current timezone label.     | ✅                     | `os_timezone`                           |
| `language`         | Language set in the device. | ✅                     | `br_lang`                               |
| `useragent`        | User-agent.                 | ✅                     | `useragent`                             |
| `viewport`         | Screen viewport.            | ✅ (iOS) / ❌ (Android) | `br_viewheight`, `br_viewwidth`         |
| `screenResolution` | Screen resolution.          | ✅                     | `dvce_screenheight`, `dvce_screenwidth` |
| `colorDepth`       | Screen color depth.         | ❌                     | `br_colordepth`                         |
| `networkUserId`    | Network user ID.            | ✅                     | `network_userid`                        |
| `domainUserId`     | Domain user ID.             | ❌                     | `domain_userid`                         |

As always, be aware of privacy when tracking [personal identifiable information](https://snowplow.io/blog/2020/09/06/user-identification-and-privacy/) such as email addresses or IP addresses. The tracker provides anonymous tracking functionality to mask certain user identifiers. Refer to the [section on anonymous tracking to learn more](/docs/sources/mobile-trackers/anonymous-tracking/).

> **Note:** The default screen resolution values are fetched from `WindowManager`, an older Android API. Screen resolution is also reported in the [platform context entity](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/), which obtains the screen size from the Android context resources. The height value will likely be higher in `dvce_screenheight` than in the context entity, as the `WindowManager` size includes the menu bar.
>
> To use the modern API throughout the tracker and standardize the screen resolution between event and entity properties, set `useContextResourcesScreenResolution(true)` flag in `SubjectConfiguration`. This flag is off by default, and available from Android tracker v6.0.3 onwards.

## Overriding autogenerated event properties

All enriched Snowplow events contain values for `user_ipaddress`, `useragent`, and `network_userid`.

Unless [anonymous tracking with server anonymisation](/docs/sources/mobile-trackers/anonymous-tracking/) is enabled, the `user_ipaddress` is automatically added to all enriched events. To manually override this without using anonymous tracking, use a `SubjectConfiguration` and set an `ipAddress` string; use an empty string to prevent IP address tracking. Alternatively, use the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/).

The `useragent` is also automatically added but it can be overriden on configuration. Snowplow pipelines provide multiple useragent-parsing [enrichments](/docs/pipeline/enrichments/available-enrichments/). To manually override the detected useragent, set a `useragent` string.

The `network_userid` is the cookie value for the event collector's third-party cookie. The cookie is named `sp` (or `micro` for Snowplow Micro pipelines). To override the collector cookie’s value with your own generated ID, set `networkUserId`. Again, anonymous tracking with server anonymisation will prevent the addition of this property to events. The `network_userid` is stored in the tracker and is kept the same until the app is deleted or the collector endpoint is changed or the cookie is expired.

All other `Subject` properties can also be manually overriden via `SubjectConfiguration` and `SubjectController`.

## Set the `Subject` properties

Initialize the tracker with a `SubjectConfiguration` like this:

**iOS:**

```swift
let subjectConfig = SubjectConfiguration()

    .userId("username")



let networkConfig = NetworkConfiguration(endpoint: "https://snowplow-collector-url.com")

Snowplow.createTracker(

    namespace: "appTracker",

    network: networkConfig,

    configurations: [subjectConfig]

)
```

See the API docs for the full [list of options](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/subjectconfiguration).

**Android (Kotlin):**

```kotlin
val subjectConfig = SubjectConfiguration()

    .userId("username")



val networkConfig = NetworkConfiguration("[http:/endpoint](https://snowplow-collector-url.com)")

Snowplow.createTracker(

    applicationContext,

    "namespace",

    networkConfig,

    subjectConfig

)
```

See the API docs for the full [list of options](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/-subject-configuration/index.html).

**Android (Java):**

```java
SubjectConfiguration subjectConfig = new SubjectConfiguration()

    .userId("username");



NetworkConfiguration networkConfig = new NetworkConfiguration(

    "https://snowplow-collector-url.com"

);

Snowplow.createTracker(

    getApplicationContext(),

    "appTracker",

    networkConfig,

    subjectConfig

);
```

See the API docs for the full [list of options](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/-subject-configuration/index.html).

***

Subject properties can be updated or added to after initialization using the `SubjectController` retrieved from the tracker:

**iOS:**

```swift
let tracker = Snowplow.defaultTracker()

tracker?.subject?.userId = ""
```

**Android (Kotlin):**

```kotlin
val tracker = Snowplow.defaultTracker

tracker?.subject?.userId = ''
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.getDefaultTracker();

tracker.getSubject().setUserId("");
```

***

---

# Configure how events are sent in the native mobile trackers
> Configure network connections, event batching, and retry policies for sending events from mobile trackers to your collector.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/configuring-how-events-are-sent/

A user interacts with your app: an event is generated and tracked using `track` method of `TrackerController`. But the event must be sent to your event collector, to enter your pipeline, before it has any value.

The tracker allows the configuration of the network connection, event sending, and buffering of events. The default configurations will be sufficient for many Snowplow users but a better fine tuning can be set through the `EmitterConfiguration`.

## Configuring the network connection

The `NetworkConfiguration` is used to specify the collector endpoint:

**iOS:**

```swift
let networkConfig = NetworkConfiguration(endpoint: "http://collector-endpoint")
```

**Android (Kotlin):**

```kotlin
val networkConfig = NetworkConfiguration("http://collector-endpoint")
```

**Android (Java):**

```java
NetworkConfiguration networkConfig = new NetworkConfiguration("http://collector-endpoint");
```

***

The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

In particular cases it can be useful to have a full control of the component in charge to send the events. This can be achieved with a custom `NetworkConnection` that will take care to send the events to the collector:

**iOS:**

```swift
let network = DefaultNetworkConnection(urlString: url, httpMethod: method)

network.emitThreadPoolSize = 20

network.byteLimitPost = 52000



let networkConfig = NetworkConfiguration(networkConnection: network)
```

In the example above we used the `DefaultNetworkConnection` but it can be used any custom component that implements the `NetworkConnection` interface.

**Android (Kotlin):**

```kotlin
val connection = OkHttpNetworkConnectionBuilder(url, applicationContext)

    .method(method)

    .emitTimeout(10)

    .build()



val networkConfig = NetworkConfiguration(connection)
```

In the example above we used the `OkHttpNetworkConnection` but it can be used any custom component that implements the `NetworkConnection` interface.

**Android (Java):**

```java
OkHttpNetworkConnection connection =

      new OkHttpNetworkConnection.OkHttpNetworkConnectionBuilder(url, getApplicationContext())

            .method(method)

            .emitTimeout(10)

            .build();



NetworkConfiguration networkConfig = new NetworkConfiguration(connection);
```

In the example above we used the `OkHttpNetworkConnection` but it can be used any custom component that implements the `NetworkConnection` interface.

***

## Persisting events with a custom EventStore

The tracker sends events asynchrounously in batches using POST requests. In case the collector is not reachable, the events are stored in an internal component called `EventStore` based on a SQLite database. The `EventStore` can be overriden with a custom one in case the developer require a different solution to persist the events before sending.

**iOS:**

```swift
let eventStore = CustomEventStore(namespace: trackerNamespace);

let emitterConfig = EmitterConfiguration()

      .eventStore(eventStore)
```

**Android (Kotlin):**

```kotlin
val eventStore: EventStore = CustomEventStore(applicationContext, trackerNamespace)



val emitterConfiguration = EmitterConfiguration()

    .eventStore(eventStore)
```

**Android (Java):**

```java
EventStore eventStore = new CustomEventStore(getApplicationContext(), trackerNamespace);



EmitterConfiguration emitterConfiguration = new EmitterConfiguration()

      .eventStore(eventStore);
```

***

In the example above the `CustomEventStore` is your implementation of the `EventStore` interface.

## What happens if an event fails to send?

The tracker has an option for setting response codes not to retry after. The intended use is for codes such as `400 Bad Request`, `401 Unauthorised`, `403 Forbidden`, `410 Gone`, `422 Unprocessable Entity`. When received in response from the Collector, the tracker doesn't retry to send events. Requests with all other 3xx, 4xx, and 5xx status codes are retried. The set of status codes for which events should be retried or not is customizable in `EmitterConfiguration`.

By default, the tracker retries sending events on all 3xx, 4xx, and 5xx status codes except the status codes indicated above. You may override the default behavior using the `customRetryForStatusCodes`. Please note that not retrying sending events to the Collector means that the events will be dropped when they fail to be sent. The `customRetryForStatusCodes` needs a dictionary that maps integers (status codes) to booleans (true for retry and false for not retry).

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()

    .customRetryForStatusCodes({403, true}) // retry sending events even if collector returns 403 status
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()

    .customRetryForStatusCodes(mapOf(

        403 to true

    ))
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()

      .customRetryForStatusCodes(new HashMap<>() {{ put(403, true); }});
```

***

Starting from version 5.5.0, you can also completely disable retrying failed requests to the collector using the `EmitterConfiguration.retryFailedRequests` configuration option. If configured, events that fail to be sent in the first request to the collector will be dropped. This may be useful in situations where it's necessary to prevent traffic spikes with many events being sent at the same time.

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()

    .retryFailedRequests(false) // don't retry any failed requests to the collector (defaults to true)
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()

    .retryFailedRequests(false) // don't retry any failed requests to the collector (defaults to true)
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()

      .retryFailedRequests(false); // don't retry any failed requests to the collector (defaults to true)
```

***

## Configuring how many events to send in one request

There are two options in the `EmitterConfiguration` that are relevant for configuring how many events are sent per request to the collector:

1. `bufferOption` – How many events to wait for before making a request. Defaults to 1.
2. `emitRange` – The maximum amount of events to send in a request. Defaults to 25.

The `bufferOption` tells the tracker how many events have to accumulate in the event store before it should make a request to the collector. There are three options: 1 (`BufferOption.single`), 10 (`BufferOption.smallGroup`), or 25 (`BufferOption.largeGroup`) events. Choosing `BufferOption.smallGroup` means that 10 events need to be tracked before the first request to the collector is made.

With very high event volumes, or when many events are buffered in the event store (e.g. if the network had been down), the `emitRange` settings come into play. This setting specifies the maximum number of events that can be sent in one request to the collector. For instance, let's say that 100 events accumulate in the event store. In case the default emit range (25) is used, the tracker will make 4 requests serially, one after the other, each with 25 events.

When the event store is empty, the emitter stops sending and waits for new events. It resumes automatically when a new event is tracked or when you call [`flush`](#manually-flushing-the-event-queue). You don't need to manage this yourself.

Configure the batch size and emit range like this:

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()

      .bufferOption(BufferOption.single)

      .emitRange(25)
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()

    .bufferOption(BufferOption.Single)

    .emitRange(25)
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()

      .bufferOption(BufferOption.Single)

      .emitRange(25);
```

***

> **Note:** Before version 6 of the iOS and Android tracker, the `bufferOption` and `emitRange` had a slightly different meaning. Events were sent right after they were tracked regardless of the `bufferOption` used. The `bufferOption` was used to specify the maximum number of events per request and the `emitRange` specified the maximum number of total events to make in parallel requests at once. For instance, if there were 100 events in the event store and `bufferOption` was set to 10 (called `defaultGroup` previously) and `emitRange` to 50, the tracker would make 5 parallel requests to the collector with 10 events each. After that it would make another 10 parallel requests to the collector with 10 events each.

## Manually flushing the event queue

The tracker sends events asynchronously and, depending on the `bufferOption`, may hold events in the event store until enough have accumulated to make a request. In some cases you want to send whatever is queued right away rather than waiting for the buffer to fill. A common example is sending any remaining events when the user logs out.

Call `flush` on the `EmitterController` to attempt to send all events currently in the event store immediately:

**iOS:**

```swift
tracker.emitter?.flush()
```

**Android (Kotlin):**

```kotlin
tracker.emitter.flush()
```

**Android (Java):**

```java
tracker.getEmitter().flush();
```

***

Sending happens asynchronously, so events may not have reached the collector by the time `flush` returns. Events are persisted in the event store before sending, so they are not lost if the app is closed before the request completes; they are sent on the next `flush` or when the tracker next sends events.

> **Note:** `flush` is available on the iOS tracker and, from version 6.4.0, on the Android tracker.

## Automatic clean up of the event store

> **Note:** This feature was introduced in version 6.0.0 of the iOS and Android trackers.

Under some situations events may keep accumulating in the event store without the tracker being able to send them at all. For instance, this may happen in case the user has an ad blocker installed or in case they permanently use the app offline.

If events accumulated in the event store without any limits, the size of the event store would keep getting bigger, potentially causing performance issues or app crashes. To prevent this, the tracker automatically removes old events from the event store. It removes old events based on two criteria that are configurable using `EmitterConfiguration`:

1. Maximum event store size (`maxEventStoreSize`) – in case the number of events surpasses this threshold, the oldest events will be removed until the number of events is under the threshold. Defaults to 1000.
2. Maximum event age (`maxEventStoreAge`) – events older than this threshold are removed. Defaults to 30 days.

The clean up is triggered before each emit attempt – before sending events to the collector.

You can configure the properties as follows.

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()

      .maxEventStoreSize(1000) // events

      .maxEventStoreAge(TimeInterval(60 * 60 * 24 * 30)) // 30 days
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()

    .maxEventStoreSize(1000)

    .maxEventStoreAge(30.toDuration(DurationUnit.DAYS))
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()

      .maxEventStoreSize(1000)

      .maxEventStoreAge(Duration.ofDays(30).toKotlinDuration());
```

***

---

# Global context for mobile trackers
> Apply context entities declaratively to all or specific subsets of tracked events using global context generators.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/custom-tracking-using-schemas/global-context/

As well as [adding context entities](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) to each individual event, it is possible to add context entities in a declarative way, so that they are applied to all (or a subset of) events within an application.

This can be done at tracker setup by providing a `GlobalContextConfiguration`. The logic for each global context entity is held within a `GlobalContext` generator. Multiple `GlobalContext` can be provided to the `GlobalContextConfiguration`, along with an identifying name or tag.

This example code shows the simplest kind of `GlobalContext` configuration: the same (static) self-describing JSON entity will be attached to every event tracked.

**iOS:**

```swift
// context entity to add to all events

let staticContext = SelfDescribingJson(schema: "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0", andData: ["key": "staticExample"])



// create a GlobalContext instance with the entity as a static context

let staticGlobalContext = GlobalContext(staticContexts: [staticContext])



// create a GlobalContextsConfiguration and assign the GlobalContext with a unique tag identifier

let globalContextsConfig = GlobalContextsConfiguration()

    .contextGenerators(["staticExampleTag": staticGlobalContext])



// pass the configuration when creating a new tracker

let tracker = Snowplow.createTracker(namespace: ..., network: ..., configurations: [..., globalContextsConfig])
```

**Android (Kotlin):**

```kotlin
// context entity to add to all events

val staticContext = SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    mapOf(

        "key" to "staticExample"

    )

)

// create a GlobalContext instance with the entity as a static context

val staticGlobalContext = GlobalContext(listOf(staticContext))



// create a GlobalContextsConfiguration and assign the GlobalContext with a unique tag identifier

val globalContextsConfig = GlobalContextsConfiguration(

    mutableMapOf(

        "staticExampleTag" to staticGlobalContext

    )

)



// pass the configuration when creating a new tracker

Snowplow.createTracker(applicationContext, "namespace", networkConfiguration, globalContextsConfig)
```

**Android (Java):**

```java
// context entity to add to all events

SelfDescribingJson staticContext = new SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    new HashMap<String, String>() {{ put("key", "staticExample"); }}

);

// create a GlobalContext instance with the entity as a static context

GlobalContext staticGlobalContext = new GlobalContext(Arrays.asList(staticContext));



// create a GlobalContextsConfiguration and assign the GlobalContext with a unique tag identifier

GlobalContextsConfiguration globalContextsConfig = new GlobalContextsConfiguration(

    new HashMap<String, GlobalContext>() {{ put("staticExampleTag", staticGlobalContext); }}

);



// pass the configuration when creating a new tracker

Snowplow.createTracker(getApplicationContext(), "namespace", networkConfiguration, globalContextsConfig);
```

***

A generator can also be added at run-time using the method `add` like in the following example. This is possible even if no `GlobalContextConfiguration` was originally provided when the tracker was created.

**iOS:**

```swift
tracker.globalContexts?.add(tag: "staticExampleTag", contextGenerator: staticGlobalContext)
```

**Android (Kotlin):**

```kotlin
tracker.globalContexts.add("staticExampleTag", staticGlobalContext);
```

**Android (Java):**

```java
tracker.getGlobalContexts().add("staticExampleTag", staticGlobalContext);
```

***

Each context generator is associated with a tag string. The tag can be used to remove a generator at runtime using the method `remove` like in the following example.

**iOS:**

```swift
tracker.globalContexts.remove(tag: "staticExampleTag")
```

**Android (Kotlin):**

```kotlin
tracker.globalContexts.remove("staticExampleTag")
```

**Android (Java):**

```java
tracker.getGlobalContexts().remove("staticExampleTag");
```

***

It returns `nil`/`null` in case there aren’t generators registered with the specified tag, otherwise it returns the removed `GlobalContext` instance.

## Configuring the Global Context logic

An entity can be an immutable static entity, or a dynamic entity based off the event received. Also, the entity can be added either to all events, or conditionally to a subset of events by type or schema. The specific objects passed to the `GlobalContext` generator set how the global context entity will be applied, as described in the table below:

**iOS:**

|             | All events             | By event type                       | By event schema                       |
| ----------- | ---------------------- | ----------------------------------- | ------------------------------------- |
| **Static**  | `[SelfDescribingJson]` | `[SelfDescribingJson], FilterBlock` | `[SelfDescribingJson], SchemaRuleSet` |
| **Dynamic** | `GeneratorBlock`       | `GeneratorBlock, FilterBlock`       | `GeneratorBlock, SchemaRuleSet`       |

**Android (Kotlin):**

|             | All events                 | By event type                                | By event schema                           |
| ----------- | -------------------------- | -------------------------------------------- | ----------------------------------------- |
| **Static**  | `List<SelfDescribingJson>` | `List<SelfDescribingJson>, FunctionalFilter` | `List<SelfDescribingJson>, SchemaRuleSet` |
| **Dynamic** | `FunctionalGenerator`      | `FunctionalGenerator, FunctionalFilter`      | `FunctionalGenerator, SchemaRuleSet`      |

**Android (Java):**

|             | All events                 | By event type                                | By event schema                           |
| ----------- | -------------------------- | -------------------------------------------- | ----------------------------------------- |
| **Static**  | `List<SelfDescribingJson>` | `List<SelfDescribingJson>, FunctionalFilter` | `List<SelfDescribingJson>, SchemaRuleSet` |
| **Dynamic** | `FunctionalGenerator`      | `FunctionalGenerator, FunctionalFilter`      | `FunctionalGenerator, SchemaRuleSet`      |

***

If the Global Context logic becomes too complex, consider using the `ContextGenerator` interface/protocol instead.

### Static entities

This is useful in cases where the entity is static and it's always the same. A classic case is a contextual information like a user identifier that doesn't change during the tracking.

**iOS:**

```swift
let staticContext = SelfDescribingJson(schema: "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0", andData: ["key": "staticExample"])
```

**Android (Kotlin):**

```kotlin
val staticContext = SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    mapOf(

        "key" to "staticExample"

    )

)
```

**Android (Java):**

```java
SelfDescribingJson staticContext = new SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    new HashMap<String, String>() {{ put("key", "staticExample"); }}

);
```

***

### Dynamic entities

A context generator callback (`FunctionalGenerator` on Android) returns an array of self describing JSONs, representing entities. They are evaluated each time an event is sent, and so can be used to send an entity based off event information. The `InspectableEvent` is an interface that exposes internal data of the processed event: name, schema and payload.

**iOS:**

```swift
let contextGenerator = GlobalContext(generator: {  event in

    return [

        SelfDescribingJson(schema: "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0", andData: ["eventName": event.schema!])

    ]

})
```

**Android (Kotlin):**

```kotlin
val contextGenerator = GlobalContext(object : FunctionalGenerator() {

    override fun apply(event: InspectableEvent): List<SelfDescribingJson> {

        return listOf(SelfDescribingJson(

            "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

            mapOf(

                    "eventName" to event.schema

            )

        ))

    }

})
```

**Android (Java):**

```java
GlobalContext contextGenerator = new GlobalContext(new FunctionalGenerator() {

    @Nullable

    @Override

    public List<SelfDescribingJson> apply(@NonNull InspectableEvent event) {

        return Arrays.asList(new SelfDescribingJson(

                "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

                new HashMap<String, String>() {{ put("eventName", event.getSchema()); }}

        ));

    }

});
```

***

### Conditional entities

The previous examples described the generation of entities that are associated with every event. However, there are cases where the contexts should only be applied to certain events.

#### Filter Callback

A filter callback is used to discriminate between events so we can attach global contexts only to certain events.

**iOS:**

```swift
let filteredGC = GlobalContext(

    staticContexts: [

        SelfDescribingJson(schema: "iglu:com.snowplowanalytics.snowplow/test_sdj/jsonschema/1-0-1", andData: [

            "key": "value"

        ])

    ],

    filter: { event in

        // this will add the entity to all Structured events

        return "se" == event?.name

    })

tracker.add(filteredGC, tag: "tag1")
```

**Android (Kotlin):**

```kotlin
val staticContext = SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    mapOf(

        "key" to "staticExample"

    )

)

val staticGlobalContext =

    GlobalContext(listOf(staticContext), object : FunctionalFilter() {

        override fun apply(event: InspectableEvent): Boolean {

            // this will add the entity to all Structured events

            return event.name === "se"

        }

    })
```

**Android (Java):**

```java
SelfDescribingJson staticContext = new SelfDescribingJson(

        "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

        new HashMap<String, String>() {{ put("key", "staticExample"); }}

);

GlobalContext staticGlobalContext = new GlobalContext(Arrays.asList(staticContext), new FunctionalFilter() {

    @Override

    public boolean apply(@NonNull InspectableEvent event) {

        // this will add the entity to all Structured events

        return event.getName() == "se";

    }

});
```

***

#### Ruleset Provider

A ruleset provider is used when you want to attach a global context entity to certain events based on the schema URI.

A ruleset provider has a ruleset which has a list of allowed schemas and a list of denied schemas. Both lists contain Iglu URIs which can be modified based on some syntactic rules.

In this example, the ruleset provider will attach the generated entities (as described in the previous section) to events with the schema `iglu:com.acme.*/*/jsonschema/*-*-*`, but not to `iglu:com.acme.marketing/*/jsonschema/*-*-*`.

**iOS:**

```swift
let allowed = "iglu:com.snowplowanalytics.*/*/jsonschema/*-*-*"

let denied = "iglu:com.snowplowanalytics.mobile/*/jsonschema/*-*-*"



let ruleset = SchemaRuleset(allowedList: [allowed], andDeniedList: [denied])

let rulesetGC = GlobalContext(staticContexts:[

  SelfDescribingJson(schema:"iglu:com.snowplowanalytics.snowplow/test_sdj/jsonschema/1-0-1", andData:["key": "value"])

], ruleset:ruleset)



tracker.globalContexts?.add(tag: "tag1", contextGenerator: rulesetGC)
```

**Android (Kotlin):**

```java
val allowed = "iglu:com.snowplowanalytics.*/*/jsonschema/*-*-*"

val denied = "iglu:com.snowplowanalytics.mobile/*/jsonschema/*-*-*"



val staticContext = SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    mapOf(

        "key" to "staticExample"

    )

)

val staticGlobalContext = GlobalContext(

    listOf(staticContext),

    buildRuleSet(listOf(allowed), listOf(denied))

)
```

**Android (Java):**

```java
String allowed = "iglu:com.snowplowanalytics.*/*/jsonschema/*-*-*";

String denied = "iglu:com.snowplowanalytics.mobile/*/jsonschema/*-*-*";



SelfDescribingJson staticContext = new SelfDescribingJson(

    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

    new HashMap<String, String>() {{ put("key", "staticExample"); }}

);

GlobalContext staticGlobalContext = new GlobalContext(

    Arrays.asList(staticContext),

    SchemaRuleSet.buildRuleSet(Arrays.asList(allowed), Arrays.asList(denied))

);
```

***

##### Ruleset Format

Ruleset rules are the strings used to match against certain schemas, such as `iglu:com.acme/*/jsonschema/*-*-*`.

They follow the same five-part format as an Iglu URI `protocol:vendor/event_name/format/version` with the exception that a wildcard can be used to refer to all cases.

The parts of a rule are wildcarded with certain guidelines:

- asterisks cannot be used for the protocol (i.e. schemas always start with `iglu:`);
- version matching must be specified like so: `*–*–*`, where any part of the versioning can be defined, e.g. `1-*–*`, but only sequential parts can be wildcarded, e.g. `1-*-1` is invalid but `1-1–*` is valid;
- at least two parts: `com.acme.*` is valid, while `com.*` is not;
- vendors cannot be defined with non-wildcarded parts between wildcarded parts: `com.acme.*.marketing.*` is invalid, while `com.acme.*.*` is valid.

### ContextGenerator for custom logic

In case the logic for filter and generator callbacks are too complex, it’s possible to specify them in a class that implements the `ContextGenerator` interface/protocol.

In this case the logic for filtering and generation is encapsulated behind a context generator class.

**iOS:**

```swift
class GlobalContextGenerator : ContextGenerator {



    func filter(from event: InspectableEvent) -> Bool {

        return true

    }



    func generator(from event: InspectableEvent) -> [SelfDescribingJson]? {

        return [

            SelfDescribingJson(schema:"iglu:com.snowplowanalytics.snowplow/test_sdj/jsonschema/1-0-1", andData:["key": "value"]),

        ]

    }

}
```

It can be passed to the tracker as usual:

```swift
let contextGeneratorGC = GlobalContext(contextGenerator: GlobalContextGenerator())

tracker.globalContexts?.add(tag: "tag1", contextGenerator: contextGeneratorGC)
```

**Android (Kotlin):**

```kotlin
val staticGlobalContext = GlobalContext(object : ContextGenerator {

    override fun generateContexts(event: InspectableEvent): List<SelfDescribingJson> {

        return listOf(SelfDescribingJson(

            "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

            mapOf(

                    "eventName" to event.schema

            )

        ))

    }



    override fun filterEvent(event: InspectableEvent): Boolean {

        return true

    }

})
```

**Android (Java):**

```java
GlobalContext staticGlobalContext = new GlobalContext(new ContextGenerator() {

    @NonNull

    @Override

    public List<SelfDescribingJson> generateContexts(@NonNull InspectableEvent event) {

        return Arrays.asList(new SelfDescribingJson(

                "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",

                new HashMap<String, String>() {{ put("eventName", event.getSchema()); }}

        ));

    }



    @Override

    public boolean filterEvent(@NonNull InspectableEvent event) {

        return true;

    }

});
```

***

---

# Track custom events with the native mobile trackers
> Track custom self-describing events and attach context entities using JSON schemas in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/custom-tracking-using-schemas/

Self-describing (self-referential) JSON schemas are at the core of Snowplow tracking. Read more about them [here](/docs/fundamentals/schemas/). They allow you to track completely customised data, and are also used internally throughout Snowplow pipelines.

In all our trackers, self-describing JSON are used in two places. One is in the `SelfDescribing` event type that wraps custom self-describing JSONs for sending. The second use is to attach entities to any tracked event. The entities can describe the context in which the event happen or provide extra information to better describe the event.

## Tracking a custom event (SelfDescribing)

You may wish to track events in your app which are not directly supported by Snowplow and which structured event tracking does not adequately capture. Your event may have more than the five fields offered by Structured events, or its fields may not fit into the category-action-label-property-value model. The solution is Snowplow’s self-describing events. Self-describing events are a [data structure based on JSON Schemas](/docs/fundamentals/schemas/) and can have arbitrarily many fields.

**iOS:**

```swift
let data = ["targetUrl": "http://a-target-url.com"];

let event = SelfDescribing(schema: "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", payload: data)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val data = mapOf(

    "targetUrl" to "http://a-target-url.com"

)

val json = SelfDescribingJson(

    "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",

    data

)

val event = SelfDescribing(json)
```

**Android (Java):**

```java
Map<String, String> data = new HashMap<>();

data.put("targetUrl", "http://a-target-url.com");

SelfDescribingJson json = new SelfDescribingJson("iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", data);

SelfDescribing event = new SelfDescribing(json);



tracker.track(event);
```

***

A Self Describing event is a [self-describing JSON](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/). It has two fields:

- A `data` field, containing the properties of the event
- A `schema` field, containing the location of the JSON schema against which the `data` field should be validated.

See the full configuration and parameter options for all these classes and methods in the API docs - [Android](https://snowplow.github.io/snowplow-android-tracker/index.html) and [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/).

## Tracking a custom entity

Custom entities can be used to augment any standard Snowplow event type with additional data. Each custom context is an array of self-describing JSON following the same pattern as a self describing event. As with self describing events, if you want to create your own custom entity, you must create a JSON schema.

Note: Even if only one custom entity is being attached to an event, it still needs to be wrapped in an array.

Here are two examples of schema for custom entities. One describes a screen:

```json
{

    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

    "self": {

        "vendor": "com.example",

        "name": "screen",

        "format": "jsonschema",

        "version": "1-0-1"

    },

    "type": "object",

    "properties": {

        "screenType": {

            "type": "string"

        },

        "lastUpdated": {

            "type": "string"

        }

    }

    "required": ["screenType", "lastUpdated"],

    "additionalProperties": false

}
```

and the other describes a user on that screen:

```json
{

    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",

    "self": {

        "vendor": "com.example",

        "name": "user",

        "format": "jsonschema",

        "version": "2-0-0"

    },

    "type": "object",

    "properties": {

        "user": {

            "type": "string"

        }

    }

    "required": ["user"],

    "additionalProperties": false

}
```

They can be used in the tracker to provide more context to specific events (e.g. ScreenView event).

**iOS:**

```swift
let event = ScreenView(name: "DemoScreenName")

event.entities.add(

    SelfDescribingJson(schema: "iglu:com.example/screen/jsonschema/1-0-1",

        andDictionary: [

             "screenType": "test",

             "lastUpdated": "2021-06-11"

        ])!)

event.entities.add(

    SelfDescribingJson(schema: "iglu:com.example/user/jsonschema/2-0-0",

        andDictionary: [

             "userType": "tester"

        ])!)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ScreenView("screen")

event.entities.add(

    SelfDescribingJson(

        "iglu:com.example/screen/jsonschema/1-2-1",

        mapOf(

            "screenType" to "test",

            "lastUpdated" to "2021-06-11"

        )

    )

)

event.entities.add(

    SelfDescribingJson(

        "iglu:com.example/user/jsonschema/2-0-0",

        mapOf(

            "userType" to "tester"

        )

    )

)



tracker.track(event)
```

**Android (Java):**

```java
ScreenView event = new ScreenView("screen");

event.getEntities().add(

    new SelfDescribingJson("iglu:com.example/screen/jsonschema/1-2-1",

        new HashMap<String, String>() {{

            put("screenType", "test");

            put("lastUpdated", "2021-06-11");

        }})

);

event.getEntities().add(

    new SelfDescribingJson("iglu:com.example/user/jsonschema/2-0-0",

        new HashMap<String, String>() {{

            put("userType", "tester");

        }})

);



tracker.track(event);
```

***

---

# Example applications for the native mobile trackers
> Explore example Android and iOS applications demonstrating implementation of the Snowplow mobile trackers.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/demos/

Check out the following example applications that can be used as a reference for how to use the tracker, or as a starting point for your own implementation.

## Android Tracker Demos

- [Java](https://github.com/snowplow/snowplow-android-tracker/tree/master/snowplow-demo-java)
- [Kotlin](https://github.com/snowplow/snowplow-android-tracker/tree/master/snowplow-demo-kotlin)
- [Jetpack Compose](https://github.com/snowplow/snowplow-android-tracker/tree/master/snowplow-demo-compose)

## iOS Tracker Demos

The [examples repo](https://github.com/snowplow-industry-solutions/snowplow-ios-tracker-examples) contains examples of some of the ways you can use the iOS Tracker, including:

- A Swift SPM app
- A Swift Cocoapods app
- An ObjC Cocoapods app
- Iglu Central app in SwiftUI

---

# Track events from hybrid apps with the native mobile trackers
> Track events from both native mobile code and embedded WebViews in hybrid apps using unified session tracking.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/hybrid-apps/

> **Info:** This feature is available since v4. To use the Web plugin, you will need v6.1+ of the iOS or Android tracker.

Hybrid apps are mobile apps that in addition to a native interface, provide part of the UI through an embedded WebView. Snowplow events are tracked from both the native code (e.g., written in Swift or Kotlin) as well as the WebView (in JavaScript). Our goal is to have both events tracked from the native code as well as the WebView share the same session and appear as tracked with the same tracker.

## Event forwarding

We recommend using the Web tracker (v4.3+) to forward all Web events to the mobile tracker.

1. Implement the Snowplow iOS or Android tracker.

2. Implement the [Snowplow Web/JavaScript tracker](/docs/sources/web-trackers/) in the WebView in your app. Make sure to include the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/).

3. Subscribe to WebView event messages.

   **iOS:**

   ```swift
   // Pass in a WKWebViewConfiguration

   Snowplow.subscribeToWebViewEvents(with: webView.configuration)
   ```

   **Android (Kotlin):**

   ```kotlin
   // Pass in a WebView object

   Snowplow.subscribeToWebViewEvents(webView);
   ```

   **Android (Java):**

   ```java
   // Pass in a WebView object

   Snowplow.subscribeToWebViewEvents(webView);
   ```

   ***

4. Track events as usual.

The Web tracker will automatically intercept all web events and forward them to the mobile tracker. The forwarded events will have the tracker version from Web, e.g. "js-4.1.0", but will otherwise be tracked like the mobile events. They may contain additional information not present in the native mobile events, such as a browser useragent string or URL, or Web context entities e.g. the [WebPage entity](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity).

The forwarded events are filtered out of the Web tracker event queue so that they are not tracked twice.

The WebView plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## WebView Tracker

If you don't want to implement a Web tracker in your WebView, you can use the [Snowplow WebView tracker](/docs/sources/webview-tracker/) directly. This could be suitable if you only want to track a small number of events from the WebView. We recommend event forwarding for most use cases.

1. Implement the Snowplow iOS or Android tracker.

2. Implement the [Snowplow WebView tracker](/docs/sources/webview-tracker/) in the WebView in your app.

3. Subscribe to WebView event messages.

   **iOS:**

   ```swift
   // Pass in a WKWebViewConfiguration

   Snowplow.subscribeToWebViewEvents(with: webView.configuration)
   ```

   **Android (Kotlin):**

   ```kotlin
   // Pass in a WebView object

   Snowplow.subscribeToWebViewEvents(webView);
   ```

   **Android (Java):**

   ```java
   // Pass in a WebView object

   Snowplow.subscribeToWebViewEvents(webView);
   ```

   ***

4. Manually track [WebView events](/docs/sources/webview-tracker/).

All event types can be tracked with WebView tracker v0.3.0+, but it requires more work than using the event forwarding.

---

# Native mobile tracker SDKs
> Use Snowplow's native mobile trackers to collect behavioral data from Android, iOS, macOS, tvOS, and watchOS applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/

Use our Snowplow [Android](https://github.com/snowplow/snowplow-android-tracker) and [iOS](https://github.com/snowplow/snowplow-ios-tracker) trackers to collect data from Android, iOS, macOS, tvOS and watchOS apps.

---

# Install and initialize the native mobile trackers
> Install the Snowplow mobile tracker SDK for iOS or Android and configure it to track events from your mobile application.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/installation-and-set-up/

**iOS:**

The Snowplow iOS Tracker SDK supports iOS 11.0+, macOS 10.13+, tvOS 12.0+, watchOS 6.0+, and visionOS 1.0+. It can be used both in Swift as well as in Objective-C code.

## Installing

The iOS Tracker SDK can be installed using various dependency managers.

**Swift Package Manager** (Recommended)

To install Snowplow Tracker with SPM:

1. In Xcode, select File > Swift Packages > Add Package Dependency.
2. Add the url where to download the library: <https://github.com/snowplow/snowplow-ios-tracker>

**Cocoapods**

To install Snowplow Tracker with Cocoapods:

1. Make sure that Cocoapods is installed on your system and correctly configured for your app.

2. Add the iOS Tracker SDK among the dependencies of your `Podfile`:

   ```ruby
   pod 'SnowplowTracker', '~> 6.0'
   ```

3. Run the command `pod install` to add the tracker to your app project.

> **Note:** Support for installing the tracker using Carthage was dropped in version 5 of the tracker.

## Setting up

Once the tracker SDK is correctly set as a dependency in your app project you have to instrument the tracker:

1. In your application delegate `AppDelegate.swift` add `import SnowplowTracker`.

2. In the `application(_:didFinishLaunchingWithOptions:)` method, set up the SDK as follows:

   ```swift
   let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")
   ```

   The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

3. It creates a tracker instance which can be used to track events like this:

   ```swift
   let event = ScreenView(name: "screen_name")

   tracker.track(event)
   ```

   If you prefer to access the tracker when the reference is not directly accessible, you can use the `defaultTracker` :

   ```swift
   Snowplow.defaultTracker()?.track(event)
   ```

You can override the default configuration with a fine grained configuration when you create the tracker. See the [API docs](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/trackerconfiguration/) for more details.

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {

  TrackerConfiguration()

      .appId("appId")

      .devicePlatform(.mobile)

      .base64Encoding(true)

      .logLevel(.off)

      .loggerDelegate(nil)

      .applicationContext(true)

      .platformContext(true)

      .platformContextProperties(nil)

      .platformContextRetriever(nil)

      .geoLocationContext(false)

      .sessionContext(true)

      .deepLinkContext(true)

      .screenContext(true)

      .screenViewAutotracking(true)

      .screenEngagementAutotracking(true)

      .lifecycleAutotracking(true)

      .installAutotracking(true)

      .exceptionAutotracking(true)

      .diagnosticAutotracking(false)

      .userAnonymisation(false)

      .immersiveSpaceContext(true)

      .advertisingIdentifierRetriever(nil)

  SessionConfiguration(

      foregroundTimeout: Measurement(value: 30, unit: .minutes),

      backgroundTimeout: Measurement(value: 30, unit: .minutes)

  )

      .continueSessionOnRestart(false)

}
```

The `createTracker` method allows the creation of multiple trackers in the same app. The `namespace` field lets you distinguish events sent by a specific tracker instance. It is mandatory even when the app uses just a single tracker instance like in the example above.

> **Note:** The trackers created with the above method are configured "locally" only. To create a tracker where the configuration can be updated through downloaded files, read [this page](/docs/sources/mobile-trackers/remote-configuration/) about remote configuration.

The [Examples Github repository](https://github.com/snowplow-industry-solutions/snowplow-ios-tracker-examples) includes demo apps for Swift and Objective-C covering the most popular dependencies managers. They are provided as simple reference apps to help you set up the tracker.

**Android (Kotlin):**

The Android Tracker SDK supports Android 5 (**API level 21+**).

## Installing

The Android Tracker SDK can be installed using Gradle.

**Gradle**

Add into your `build.gradle` file:

```gradle
dependencies {

  ...

  implementation 'com.snowplowanalytics:snowplow-android-tracker:6.+'

  ...

}
```

No other dependencies are required to track events. However, some **optional** dependencies can be added:

- `InstallReferrer` to enable the referrer context entity of the [`ApplicationInstall` event](/docs/sources/mobile-trackers/tracking-events/installation-tracking/).
- `Play Services` dependencies for [tracking the app set ID and AAID](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/).

## Setting up

Once the tracker SDK is correctly set as a dependency in your app project you have to instrument the tracker:

1. In your `Application` subclass, set up the SDK as follows:

   ```kotlin
   val tracker = Snowplow.createTracker(

       applicationContext, // Android context (LocalContext.current in Compose apps)

       "appTracker", // namespace

       "https://snowplow-collector-url.com" // Event collector URL

   )
   ```

   The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

2. It creates a tracker instance which can be used to track events like this:

   ```kotlin
   val event = ScreenView("screen_name")

   tracker.track(event)
   ```

If you prefer to access the tracker when the reference is not directly accessible, you can use the `defaultTracker` :

```kotlin
Snowplow.defaultTracker?.track(event)
```

You can override the default configuration with a fine grained configuration when you create the tracker. See the [Android API docs](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/index.html) for more details.

```kotlin
val networkConfig = NetworkConfiguration(

    "https://snowplow-collector-url.com",

    HttpMethod.POST

)

val trackerConfig = TrackerConfiguration("appId")

    .devicePlatform(DevicePlatform.Mobile)

    .base64encoding(true)

    .logLevel(LogLevel.OFF)

    .loggerDelegate(null)

    .applicationContext(true)

    .platformContext(true)

    .geoLocationContext(false)

    .sessionContext(true)

    .deepLinkContext(true)

    .screenContext(true)

    .screenViewAutotracking(true)

    .screenEngagementAutotracking(true)

    .lifecycleAutotracking(true)

    .installAutotracking(true)

    .exceptionAutotracking(true)

    .diagnosticAutotracking(false)

    .userAnonymisation(false)

    .platformContextProperties(null)

    .platformContextRetriever(null)

val sessionConfig = SessionConfiguration(

    TimeMeasure(30, TimeUnit.SECONDS),

    TimeMeasure(30, TimeUnit.SECONDS)

)

    .continueSessionOnRestart(false)

createTracker(

    applicationContext,

    "appTracker",

    networkConfig,

    trackerConfig,

    sessionConfig

)
```

> **Note:** The trackers created with the above method are configured "locally" only. To create a tracker where the configuration can be updated through downloaded files, read [this page](/docs/sources/mobile-trackers/remote-configuration/) about remote configuration.

The [Android tracker Github repository](https://github.com/snowplow/snowplow-android-tracker) includes demo apps in Java, Kotlin, and Kotlin with Jetpack Compose. They are provided as simple reference apps to help you set up the tracker.

## Using the tracker with R8 optimization

Depending on your app configuration, you may need to add ProGuard rules to prevent the R8 compiler removing code needed for tracker function. Fetching certain [platform context](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/) properties - AAID and app set ID - uses reflection. To include the necessary classes, add the following rules to the app's `proguard-rules.pro` file.

```text
# Reflection for the appSetId

-keep class com.google.android.gms.appset.AppSet { *; }

-keep class com.google.android.gms.appset.AppSetIdInfo { *; }

-keep class com.google.android.gms.internal.appset.zzr { *; }

-keep class com.google.android.gms.tasks.Tasks { *; }



# Reflection for the AAID (AndroidIdfa)

-keep class com.google.android.gms.ads.identifier.** { *; }
```

**Android (Java):**

The Android Tracker SDK supports Android 5 (**API level 21+**).

## Installing

The Android Tracker SDK can be installed using Gradle.

**Gradle**

Add into your `build.gradle` file:

```gradle
dependencies {

  ...

  implementation 'com.snowplowanalytics:snowplow-android-tracker:6.+'

  ...

}
```

No other dependencies are required to track events. However, some **optional** dependencies can be added:

- `InstallReferrer` to enable the referrer context entity of the [`ApplicationInstall` event](/docs/sources/mobile-trackers/tracking-events/installation-tracking/).
- `Play Services` dependencies for [tracking the app set ID and AAID](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/).

## Setting up

Once the tracker SDK is correctly set as a dependency in your app project you have to instrument the tracker:

1. In your `Application` subclass, set up the SDK as follows:

   ```java
   TrackerController tracker = Snowplow.createTracker(

         getApplicationContext(), // Android context

         "appTracker", // namespace

         "https://snowplow-collector-url.com" // Event collector URL

   );
   ```

   The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

2. It creates a tracker instance which can be used to track events like this:

   ```java
   Event event = new ScreenView("screen_name");

   tracker.track(event);
   ```

If you prefer to access the tracker when the reference is not directly accessible, you can use the `defaultTracker` :

```java
Snowplow.getDefaultTracker().track(event);
```

You can override the default configuration with a fine grained configuration when you create the tracker. See the [Android API docs](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/index.html) for the `Configuration` classes to see all the options and defaults.

```java
NetworkConfiguration networkConfig = new NetworkConfiguration(

    "https://snowplow-collector-url.com",

    HttpMethod.POST

);

TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .base64encoding(false)

    .sessionContext(true)

    .platformContext(true)

    .lifecycleAutotracking(true)

    .screenViewAutotracking(true)

    .screenContext(true)

    .applicationContext(true)

    .exceptionAutotracking(true)

    .installAutotracking(true)

    .userAnonymisation(false)

    .logLevel(LogLevel.OFF);

SessionConfiguration sessionConfig = new SessionConfiguration(

    new TimeMeasure(30, TimeUnit.SECONDS),

    new TimeMeasure(30, TimeUnit.SECONDS)

)

    .continueSessionOnRestart(false);

Snowplow.createTracker(

    getApplicationContext(),

    "appTracker",

    networkConfig,

    trackerConfig,

    sessionConfig

);
```

> **Note:** The trackers created with the above method are configured "locally" only. To create a tracker where the configuration can be updated through downloaded files, read [this page](/docs/sources/mobile-trackers/remote-configuration/) about remote configuration.

The [Android tracker Github repository](https://github.com/snowplow/snowplow-android-tracker) includes demo apps in Java, Kotlin, and Kotlin with Jetpack Compose. They are provided as simple reference apps to help you set up the tracker.

## Using the tracker with R8 optimization

Depending on your app configuration, you may need to add ProGuard rules to prevent the R8 compiler removing code needed for tracker function. Fetching certain [platform context](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/) properties - AAID and app set ID - uses reflection. To include the necessary classes, add the following rules to the app's `proguard-rules.pro` file.

```text
# Reflection for the appSetId

-keep class com.google.android.gms.appset.AppSet { *; }

-keep class com.google.android.gms.appset.AppSetIdInfo { *; }

-keep class com.google.android.gms.internal.appset.zzr { *; }

-keep class com.google.android.gms.tasks.Tasks { *; }



# Reflection for the AAID (AndroidIdfa)

-keep class com.google.android.gms.ads.identifier.** { *; }
```

***

---

# Migration guides for the native mobile trackers
> Upgrade to newer versions of the iOS and Android mobile trackers using version-specific migration guides.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/

This section contains information to help you upgrade to newer versions of the mobile trackers.

---

# Migration guide for Snowplow Android Tracker SDK from version 1.x to 2.0
> Migrate your Android tracker instrumentation from version 1.x to 2.0 with updated configuration and API changes.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-android-tracker-sdk-from-version-1-x-to-2-0/

This section describes the main changes when converting an applications instrumentation of the Snowplow Android tracker SDK from version 1.x to version 2.0.

## The new approach

The main objective of this new release is a revision of the public API. Currently, the tracker configuration requires the creation of the Tracker and Emitter components. Optionally, the Subject component can be configured as well. Then both need to be passed to the Tracker component. The version 2.0 simplifies this process, letting the developer to specify the configuration with Configuration objects and passing them to a `createTracker` method able to return the tracker ready to use, without any other extra steps.

### Setup of the tracker

The entry point to setup the tracker is now the `Snowplow` static class and the `createTracker` methods. They need a namespace string which is now mandatory. The namespace string uniquely identifies the tracker in the app. It's important to note that all the events not sent to the collector but stored in the tracker are attached to the namespace string. This means that if a new configuration uses a different namespace, all those unsent event will still stored in the tracker, in a sort of zombie state, with no way to send them to any collector. To send them you need to reuse the original namespace string or delete them.

Fine tuning of the tracker is now possible with Configuration classes.

These are the classes now available:

- `NetworkConfiguration`: to configure network connection with the Snowplow collector.
- `TrackerConfiguration`: to configure contexts and automatic events of the tracker, and general behavior.
- `SessionConfiguration`: to configure session behavior.
- `EmitterConfiguration`: to fine tune about how the tracker sends events to the collector.
- `SubjectConfiguration`: to specify details to send with events about the user and the platform.
- `GdprConfiguration`: to configure the GDPR context.
- `GlobalContextsConfiguration`: to configure the GlobalContexts feature to dynamically send created contexts with some selected events.

They replace many of the settings previously set on the builders for the Tracker, Subject and Emitter components.

An example configuration:

```java
TrackerController initAndroidTracker(Context context, String trackerNamespace) {

        NetworkConfiguration networkConfiguration = new NetworkConfiguration("https://snowplow-collector-url.com");

        TrackerConfiguration trackerConfiguration = new TrackerConfiguration("example-of-app-id")

                .sessionContext(true)

                .platformContext(true)

                .applicationContext(true)

                .geoLocationContext(true)

                .lifecycleAutotracking(true)

                .screenViewAutotracking(true)

                .screenContext(true)

                .exceptionAutotracking(true)

                .installAutotracking(true)

                .diagnosticAutotracking(true);

        SessionConfiguration sessionConfiguration = new SessionConfiguration(

                new TimeMeasure(60, TimeUnit.SECONDS),

                new TimeMeasure(30, TimeUnit.SECONDS)

        );

        GdprConfiguration gdprConfiguration = new GdprConfiguration(

                Basis.CONSENT,

                "someId",

                "0.1.0",

                "this is a demo document description"

        );



        return Snowplow.createTracker(context,

                trackerNamespace,

                networkConfiguration,

                trackerConfiguration,

                sessionConfiguration,

                gdprConfiguration

        );

    }
```

The tracker can be controlled by the `TrackerController` (the `Tracker` class is now deprecated). Through the `TrackerController` it will be possible to update settings at runtime and get information about the tracking state. From the `TrackerController` it's also possible to access all the other controllers. This allows you to access and update settings about communication with the collector, emission of the events, session of the tracker (if enabled) and so on. In general, there is a specific controller for each area covered by a configuration. All of them are accessible through the `TrackerController`.

**Note:** Please, don't retain a reference to the sub-controllers. If you invalidate or recreate the tracker, all the sub-controller instances will no longer be active.

## Low level migration from 1.x

- Minimum supported version is API 21 (Android 5).

- Migrated from Android Support Library to AndroidX.

- Minimum OkHttpClient dependency is 4.9.1.

- `lifecycleAutotracking` (background and foreground events and indexes on sessions) are optional and off by default.

- `lifecycleAutotracking` requires `androidx.lifecycle:lifecycle-extensions` among the dependencies of the app.

- `RequestSecurity` class has been renamed `Protocol`.

- `mobileContext` property has been renamed `platformContext`.

- `DevicePlatforms` has been renamed `DevicePlatform`.

- `ReadyRequest` has been replaced by `Request` (not needed in the new API).

- `namespace` on Tracker is now mandatory for the correct operativity.

- `name` field on `ScreenView` is now mandatory.

- `EcommerceItem`s aren't forced to have the same timestamp of the `EcommerceTransaction` event.

- `SPEcommerceItem` constructor doesn't require `itemId` as it's added by the tracker when the ecommerce event is sent.

- Deprecated classes of the previous API (Tracker, Emitter, Subject, ...) are now accessible from the `internal` sub-package.

- The `internal` package is considered private API so it doesn't follow the restrictions of the [Semantic Version](https://semver.org/) policy. Although we will keep the legacy deprecated API with minimum changes until the release of the version 3.0.

- Global Contexts implementation is changed and uniformed to the one on iOS tracker.

- **For the versions 2.0 and 2.1**, the application install events report the timestamp of when installation happened in the `true_timestamp` rather than the `device_timestamp`. It caused an [issue](https://github.com/snowplow/snowplow-android-tracker/issues/462) with the `derived_timestamp`. For this reason, since the version 2.2, the application install events report the timestamp of then installation happened in the `device_timestamp` like the previous 1.x versions.

## Migrate to new Global Contexts API

- The `ContextGenerator` no longer owns the `tag` string used to identify the Global Context generator in the tracker. The `ContextGenerator` replacement is simply called `GlobalContext` which is the new generator class for the Global Contexts feature.
- The generator `GlobalContext` is added to the tracker and associated to the specified tag through the method `add` of `GlobalContextsConfiguration`.
- The classes `FilterProvider` and `RulesetProvider` have been removed. `GlobalContext` class and `ContextGenerator` protocol can be used to filter events through specific event fields or rulesets.

---

# Migration guide for Snowplow Android Tracker SDK from version 2.x to 3.0
> Migrate your Android tracker from version 2.x to 3.0 which removes the deprecated v1 API.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-android-tracker-sdk-from-version-2-x-to-3-0/

This new release doesn't introduce breaking changes with the previous 2.x version. The version 2.0 can be considered a transitory release because two different APIs were available for the developer: v1 API (deprecated) and v2 API (the new one). The version 3.0 doesn't introduce changes to the v2 API but it completely remove the v1 API. Hence, this version introduces breking changes only for the developer that have instrumented the v2.0 using the old deprecated v1 API.

In this release we've marked as deprecated:

- `PageView` event: this event has been designed for web tracking. Its role can be replaced on mobile with two different events: `ScreenView` event and `DeepLinkReceived` event. The first covers the need of tracking the screen visualized to the user by the app. The latter covers the case where the shown content is referred to an external url referral and we want to track that down, exactly like the PageView does in the web tracker.

In this release there are some breaking changes:

- The v1 API has been removed from this new version of the trackers, which means the v1 components (Tracker, Emitter, Subject, Session) used to set up the tracker are no longer available. If you have already migrated your instrumentation to the v2 API there aren't breaking changes.

- The events can be built using the constructor and builder methods. The builder classes available with v1 are no longer available. If you already build events using events' constructor suggested with the v2 API, there aren't breaking changes.

- Session callbacks have been removed. They will be reintroduced soon in one of the next minor versions. More details will be provided at the release.

- Utilities or Utils methods, available in v1 API, have been removed.

---

# Migration guide for Snowplow iOS Tracker SDK from version 1.x to 2.0
> Migrate your iOS tracker instrumentation from version 1.x to 2.0 with updated configuration and improved Swift compatibility.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-ios-tracker-sdk-from-version-1-x-to-2-0/

This section describes the main changes when converting an applications instrumentation of the Snowplow iOS tracker SDK from version 1.x to version 2.0.

## The new approach

The main objective of this new release is a revision of the public API. Currently, the tracker configuration requires the creation of the Tracker and Emitter components. Optionally, the Subject component can be configured as well. Then both need to be passed to the Tracker component. The version 2.0 simplifies this process, letting the developer to specify the configuration with Configuration objects and passing them to a `createTracker` method able to return the tracker ready to use, without any other extra steps.

The tracker has been also updated improving the compatibility with Swift. Now, the classes of the public API don't need the `SP` prefix required on Objective-C.

### Setup of the tracker

The entry point to setup the tracker is now the `Snowplow` class and the `createTracker` methods. They need a namespace string which is now mandatory. The namespace string uniquely identifies the tracker in the app. It's important to note that all the events not sent to the collector but stored in the tracker are attached to the namespace string. This means that if a new configuration uses a different namespace, all those unsent event will still stored in the tracker, in a sort of zombie state, with no way to send them to any collector. To send them you need to reuse the original namespace string or delete them.

Fine tuning of the tracker is now possible with Configuration classes.

These are the classes now available:

- `NetworkConfiguration`: to configure network connection with the Snowplow collector.
- `TrackerConfiguration`: to configure contexts and automatic events of the tracker, and general behavior.
- `SessionConfiguration`: to configure session behavior.
- `EmitterConfiguration`: to fine tune about how the tracker sends events to the collector.
- `SubjectConfiguration`: to specify details to send with events about the user and the platform.
- `GdprConfiguration`: to configure the GDPR context.
- `GlobalContextsConfiguration`: to configure the GlobalContexts feature to dynamically send created contexts with some selected events.

They replace many of the settings previously set on the builders for the Tracker, Subject and Emitter components.

An example configuration:

```swift
func initTracker(trackerNamespace: String) -> TrackerController {

        let networkConfig = NetworkConfiguration(endpoint: "https://snowplow-collector-url.com")

        let trackerConfig = TrackerConfiguration()

            .sessionContext(true)

            .platformContext(true)

            .applicationContext(true)

            .lifecycleAutotracking(true)

            .screenViewAutotracking(true)

            .screenContext(true)

            .exceptionAutotracking(true)

            .installAutotracking(true)

            .diagnosticAutotracking(true)

        let sessionConfig = SessionConfiguration(

            foregroundTimeout: Measurement(value: 60, unit: .seconds),

            backgroundTimeout: Measurement(value: 30, unit: .seconds)

        )

        let gdprConfig = GDPRConfiguration(

            basis: .consent,

            documentId: "id",

            documentVersion: "1.0",

            documentDescription: "description"

        )



        return Snowplow.createTracker(

            namespace: trackerNamespace,

            network: networkConfig,

            configurations: [trackerConfig, emitterConfig, sessionConfig, gdprConfig]);

}
```

The tracker can be controlled by the `TrackerController` (the `Tracker` class is now deprecated). Through the `TrackerController` it will be possible to update settings at runtime and get information about the tracking state. From the `TrackerController` it's also possible to access all the other controllers. This allows you to access and update settings about communication with the collector, emission of the events, session of the tracker (if enabled) and so on. In general, there is a specific controller for each area covered by a configuration. All of them are accessible through the `TrackerController`.

**Note:** Please, don't retain a reference to the sub-controllers. If you invalidate or recreate the tracker, all the sub-controller instances will no longer be active.

## Low level migration from 1.x

- iOS minimum supported version is bumped to iOS 9.0.

- `lifecycleAutotracking` (background and foreground events and indexes on sessions) are optional and off by default.

- Enum `SPProtocol` items have been renamed to `SPProtocolHttp` and `SPProtocolHttps`.

- Enum `SPRequestOptions` has been renamed `SPHttpMethod` and its items are `SPHttpMethodGet` and `SPHttpMethodPost`.

- `Snowplow` class is now the entry point to create trackers. The old `Snowplow` class containing the constants has been renamed `SPTrackerConstants`.

- `SPPrimitive` and `SPSelfDescribing` have been renamed as `SPPrimitiveAbstract` and `SPSelfDescribingAbstract`.

- `SPUnstructured` class used for self-describing events has been renamed `SPSelfDescribing`.

- The property `trueTimestamp` changed the type from `NSNumber` to `NSDate`.

- `namespace` on Tracker is now mandatory for the correct operativity.

- `name` field on `ScreenView` is now mandatory.

- `EcommerceItem`s aren't forced to have the same timestamp of the `EcommerceTransaction` event.

- `SPEcommerceItem` constructor doesn't require `itemId` as it's added by the tracker when the ecommerce event is sent.

- Deprecated classes of the previous API (SPTracker, SPEmitter, SPSubject, ...).

- **For the versions 2.0 and 2.1**, the application install events report the timestamp of when installation happened in the `true_timestamp` rather than the `device_timestamp`. It caused an [issue](https://github.com/snowplow/snowplow-ios-tracker/issues/625) with the `derived_timestamp`. For this reason, since the version 2.2, the application install events report the timestamp of then installation happened in the `device_timestamp` like the previous 1.x versions.

---

# Migration guide for Snowplow iOS Tracker SDK from version 2.x to 3.0
> Migrate your iOS tracker from version 2.x to 3.0 which removes the deprecated v1 API.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-ios-tracker-sdk-from-version-2-x-to-3-0/

This new release doesn't introduce breaking changes with the previous 2.x version. The version 2.0 can be considered a transitory release because two different APIs were available for the developer: v1 API (deprecated) and v2 API (the new one). The version 3.0 doesn't introduce changes to the v2 API but it completely remove the v1 API. Hence, this version introduces breking changes only for the developer that have instrumented the v2.0 using the old deprecated v1 API.

In this release we've marked as deprecated:

- `PageView` event: this event has been designed for web tracking. Its role can be replaced on mobile with two different events: `ScreenView` event and `DeepLinkReceived` event. The first covers the need of tracking the screen visualized to the user by the app. The latter covers the case where the shown content is referred to an external url referral and we want to track that down, exactly like the PageView does in the web tracker.

- `PushNotification` event: this event is available only on iOS tracker and it's strongly influenced by the format of push notifications in the iOS platform. It has been replaced with `MessageNotification` which is usable in both iOS and Android trackers v3.0.

In this release there are some breaking changes:

- The v1 API has been removed from this new version of the trackers, which means the v1 components (Tracker, Emitter, Subject, Session) used to set up the tracker are no longer available. If you have already migrated your instrumentation to the v2 API there aren't breaking changes.

- The events can be built using the constructor and builder methods. The builder classes available with v1 are no longer available. If you already build events using events' constructor suggested with the v2 API, there aren't breaking changes.

- Session callbacks have been removed. They will be reintroduced soon in one of the next minor versions. More details will be provided at the release.

- Utilities or Utils methods, available in v1 API, have been removed.

---

# Migration guide from version 3.x to 4.0
> Migrate to mobile tracker version 4.0 with updated remote configuration callbacks and event ID returns.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-from-version-3-x-to-4-0/

A breaking change in this release is the updated callback for remote configuration. The callback now in addition to the list of namespaces also receives the configuration state enum as an argument. Read more about the [new API here](/docs/sources/mobile-trackers/remote-configuration/).

The tracker now also returns the event ID from the `track(event)` method in `TrackerController`.

---

# Migration guide from version 4.x to 5.0
> Migrate to mobile tracker version 5.0 with internal rewrite to Swift and Kotlin while maintaining minimal API changes.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-from-version-4-x-to-5-0/

Although the trackers underwent a huge internal rewrite (from Objective-C to Swift and from Java to Kotlin), we tried to keep the API with as little breaking changes as possible.

## iOS tracker

The supported platforms have changed on the iOS tracker:

- Minimum iOS deployment target changed from 9.0 to 11.0.
- On macOS, from 10.10 to 10.13.
- On tvOS, from 9.0 to 12.0.
- On watchOS, from 2.0 to 6.0.

Additionally, there is a new way to enable tracking the IDFA identifier – instead of adding the `SNOWPLOW_IDFA_ENABLED` compiler flag, one now needs to implement a callback (`TrackerConfiguration.advertisingIdentifierRetriever`) that retrieves the value. See the [documentation for more information](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#identifier-for-advertisers-idfaaaid).

If using `EmitterConfiguration` when creating a new tracker, the default buffer option configuration changed from `single` (max 1 event per batch) to `default` (max 10 events per batch). If not using an `EmitterConfiguration`, the buffer option was set to `default` also in tracker v4.

## Android tracker

We adopted using Kotlin properties instead of Java fields for public properties of the event classes. This doesn't change the API in Kotlin. On the other hand, in Java, the properties are now accessible through getter and setter methods instead of directly as Java fields.

For example, to set the true timestamp of events in Java the API changes as follows:

```java
// v4 API:

event.trueTimestamp = 123456789L; // doesn't work anymore

// v5 API:

event.setTrueTimestamp(123456789L);
```

However, the API hasn't changed if you use the builder methods to set the properties. For example, this approach works in both the v4 and v5 tracker:

```java
// works both in v4 and v5 API:

event.trueTimestamp(123456789L);
```

If using `EmitterConfiguration` when creating a new tracker, the default buffer option configuration changed from `single` (max 1 event per batch) to `default` (max 10 events per batch). If not using an `EmitterConfiguration`, the buffer option was set to `default` also in tracker v4.

## Renaming contexts to entities

The `contexts` property in events used to assign custom context entities to events has been renamed to `entities`. The previous naming is still available but it is deprecated.

In the Android tracker, the `customContexts` property in events has been deprecated in favor of the `entities` property.

---

# Migration guide from version 5.x to 6.0
> Migrate to mobile tracker version 6.0 with changes to lifecycle tracking, event batching, and screen engagement features.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-from-version-5-x-to-6-0/

There were a few breaking changes within v6, for both iOS and Android. These are described here, with a quick summary of the rest of the changelog at the end.

## Changes to events

These changes probably won't break your app code, but they will affect the events generated.

### Lifecycle autotracking

Affects: iOS ✅ Android ✅

[Lifecycle autotracking](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/) is now on by default, for both trackers. This is because it's a prerequisite for the new [screen engagement tracking](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking), which is also on by default.

### Removed properties from platform context entity

Affects: iOS ✅ Android ❌

To comply with Apple's Privacy Manifest rules, we have removed the automatic tracking of `totalStorage` and `availableStorage` metrics from the [platform context entity](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/). We've also added a Privacy Manifest for the SDK.

To track `totalStorage` or `availableStorage`, use the new `PlatformContextRetriever` callbacks class to configure the platform context entity.

### Preventing unnecessary ScreenView event

Affects: iOS ❌ Android ✅

Previously a screen view event was tracked again by the [screen view autotracking](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) feature when the app moved to foreground. This is not expected because the screen doesn't change when the app is in background, and it is not consistent with how the screen view autotracking works on iOS. The extra event will no longer be tracked.

### Event entities API

Affects: iOS ✅ Android ❌

To standardise the behavior between trackers, we've changed how the `entities()` method of all events works on iOS. Previously, calling `event.entities(newListEntities)` replaced all the context entities currently attached to the event. Now, the new entities are appended instead.

There's no change to the behavior of the variable `entities`, so you could still replace them all using `event.entities = newListEntities`.

## Event batching

Affects: iOS ✅ Android ✅

In both trackers, we have changed how the `EmitterConfiguration` options `bufferOption` and `emitRange` are used, as well as changing the defaults. Read more about that [here](/docs/sources/mobile-trackers/configuring-how-events-are-sent/#configuring-how-many-events-to-send-in-one-request). The `BufferOption.defaultGroup` has been renamed to `BufferOption.SmallGroup`.

Network requests are now made serially. If you are using a custom `EmitterConfiguration.emitRange`, you may wish to set it to a lower value. The new default `emitRange` is 25 (down from 150).

## Non-optional tracker

Affects: iOS ✅ Android ❌

In the v5.x of the iOS tracker, `createTracker` returned an optional `TrackerController?`. This was an oversight. In v6, the iOS tracker again returns a non-optional `TrackerController`.

## `Tracker.track()` return type

Affects: iOS ✅ Android ❌

The return type of the `Tracker.track()` method has changed from `UUID?` to `UUID` on iOS. Previously, `nil` could be returned if the tracker was paused, or if the event was filtered out and not actually tracked. From v6 onwards it will always return a UUID. If the event is tracked, this will be the `eventId`.

This change was necessary as part of the comprehensive refactoring of the tracker thread model. The iOS tracker now uses a global dispatch queue. This single queue makes the tracker much safer. Almost all actions are now performed concurrently. Network requests in Emitter are still asynchronous.

## Changes to the `EventStore` interface

Affects: iOS ✅ Android ✅

A new method, `removeOldEvents()` has been added to the `EventStore` protocol on both trackers. This method is used in the new feature that deletes events from storage if they get too old (by default, 30 days). Also, if too many events collect in the `EventStore`, the older ones will be deleted (default 1000 events).

For the Android tracker, we have updated the `EventStore` interface to remove the optional types.

## Other changes

See the full changelog on Github, for [iOS](https://github.com/snowplow/snowplow-ios-tracker/releases/tag/6.0.0) and [Android](https://github.com/snowplow/snowplow-android-tracker/releases/tag/6.0.0).

### New events

The [screen engagement](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking) feature adds new events for both iOS and Android. For iOS only, there are new events (and a new demo app) for [visionOS](/docs/sources/mobile-trackers/tracking-events/visionos/). For Android, the `PageView` event has been restored, after accidental deprecation in v5.

### Cross-navigation tracking

Decorate URIs with [user and session information](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) in both trackers using the new `CrossDeviceParameterConfiguration`. This is the equivalent of cross-domain tracking for web.

### Access to `EventStore` in iOS tracker

The `EventStore` is now exposed as part of the `EmitterController`, allowing access for e.g. deleting all stored events like this `tracker?.emitter?.eventStore?.removeAllEvents()`. This was already possible on Android.

### Codable structs in the iOS tracker

The custom event and entity classes `SelfDescribing` and `SelfDescribingJson` now accept data represented using `Encodable` structs. This alllows you to define the data using typed structs, and track that directly instead of using untyped dictionaries.

### Provide custom values to platform context entity

As mentioned above, the new `PlatformContextRetriever` callbacks class allows you to override any [platform entity](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#overriding-platform-context-properties) properties. The `PlatformContextRetriever` is available for both iOS and Android.

### Emitter and network connection behavior

The Android tracker default emit timeout has been increased to 30 seconds, from 5 seconds. This setting is configured using `NetworkConfiguration`. The `timeout` configuration option has been added to the iOS tracker.

In both trackers, the internal `Emitter` constructor has been updated. The change moves the namespace and event store into the constructor, enabling removing some optionals, and makes the properties immutable and safer.

### Removed FMDB dependency for iOS

The FMDB dependency has been removed from `SQLiteEventStore`. The built-in `sqlite` methods are used instead.

---

# Migrating to the v5.4 Ecommerce package
> Migrate from deprecated ecommerce events to the new v5.4 ecommerce tracking package with structured entities and event types.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-to-new-ecommerce/

The [Ecommerce tracking package](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) was introduced in Android and iOS version 5.4.0. It provides 11 out-of-the-box event types to make it easier to thoroughly track user activity in an ecommerce store.

> **Note:** Migrating to the new ecommerce events is a breaking change for any relevant data models.

## Replacing the deprecated events

To update your existing tracking based on the now deprecated `Ecommerce` (iOS)/`EcommerceTransaction` (Android) and `EcommerceItem`/`EcommerceTransactionItem` event types, we recommend using `TransactionEvent`.

**iOS:**

Old API:

```swift
let product = EcommerceItem(sku: "sku1", price: 100, quantity: 1)

    .name("itemName")

    .category("category")

    .currency("GBP")

    .orderId("id-123")



let event = Ecommerce(orderId: "id-123", totalValue: 350, items: [product])

    .currency("GBP")

    .taxValue(10)

    .shipping(15)

    .city("London")

    .state("N/A")

    .country("UK")

    .affiliation("affiliation")



tracker.track(event)
```

New API:

```swift
let product = ProductEntity(

  id: "sku1",

  price: 100,

  quantity: 1,

  name: "itemName",

  category: "category",

  currency: "GBP"

)



let transaction = TransactionEntity(

  transactionId: "id-123",

  revenue: 350,

  products: [product],

  currency: "GBP",

  tax: 10,

  shipping: 15

)



let event = TransactionEvent(transaction: transaction, products: [product])



tracker.track(event)
```

**Android (Kotlin):**

Old API:

```kotlin
val product = EcommerceTransactionItem(sku = "sku1", price = 100.0, quantity = 1)

    .name("itemName")

    .category("category")

    .currency("GBP")

    .orderId("id-123")



val event = EcommerceTransaction(orderId = "id-123", totalValue = 350.0, items = listOf(product))

    .currency("GBP")

    .taxValue(10.0)

    .shipping(15.0)

    .city("London")

    .state("N/A")

    .country("UK")

    .affiliation("affiliation")



tracker.track(event)
```

New API:

```kotlin
val product = ProductEntity(

  id = "sku1",

  price = 100.0,

  quantity = 1,

  name = "itemName",

  category = "category",

  currency = "GBP"

)



val transaction = TransactionEntity(

  transactionId = "id-123",

  revenue = 350.0,

  products = [product],

  currency = "GBP",

  tax = 10.0,

  shipping = 15.0

)



val event = TransactionEvent(transaction, listOf(product))



tracker.track(event)
```

**Android (Java):**

Old API:

```java
EcommerceTransactionItem product = new EcommerceTransactionItem(

    "sku1", // sku

    100.0, // price

    1 // quantity

  ).name("itemName")

  .category("category")

  .currency("GBP")

  .orderId("id-123")



List<Product> products = new ArrayList<>();

products.add(product);



EcommerceTransaction event = EcommerceTransaction(

      "id-123", // orderId

      350.0, // totalValue

      products // items

  ).currency("GBP")

  .taxValue(10.0)

  .shipping(15.0)

  .city("London")

  .state("N/A")

  .country("UK")

  .affiliation("affiliation")



tracker.track(event)
```

New API:

```java
ProductEntity product = new ProductEntity(

  "sku1", // id

  100.0, // price

  1, // quantity

  "itemName", // name

  "category", // category

  "GBP", // currency

);

List<Product> products = new ArrayList<>();

products.add(product);



TransactionEntity transaction = new TransactionEntity(

  "id-123", // id

  350.0, // revenue

  products, // products

  "GBP", // currency

  10.0, // tax

  15.0 // shipping

);



TransactionEvent event = new TransactionEvent(transaction, products);



tracker.track(event)
```

***

These code snippets show only the direct API replacement. Check out the [docs page](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) for information about expanding your tracking with the other new event types.

Note that address details and affiliation are not captured in `TransactionEvent`. To track these so that they are attached to the `TransactionEvent`, we suggest creating [custom schemas](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) and adding the data as an entity. Alternatively for addresses, you could use the `CheckoutStepEvent` which has address and postcode as optional parameters.

## How the tracked events are different

### Old events

Using the old event types, an event is generated for each `EcommerceTransaction` event as well as each `EcommerceTransactionItem` attached. Therefore, tracking an `EcommerceTransaction` event with 3 items would result in 4 events being tracked. These events are not linked at all, unless the same `orderId` is provided to all the objects, adding potential challenges in data modeling.

In the tracked data, the events have `event_name` `transaction` or `transaction_item`. The data is sent within the legacy event properties `tr_x` and `ti_x`, e.g. `tr_id` for the `EcommerceTransaction` `orderId` property. All of these properties are listed [here](/docs/events/). Each one maps to a column in the warehouse table.

### New events

The new Ecommerce event types take advantage of the Snowplow entities/event context feature. Tracking `TransactionEvent` with 3 items results in a single event being tracked. The items, and the details of the transaction, are attached as entities. That single event would have one `TransactionEntity`, three `ProductEntity`, plus whichever out-of-the-box and/or custom entities are configured.

All of the new events are `SelfDescribingEvent` with `event_name` `snowplow_ecommerce_action`. None of the data is sent as legacy properties; it is all within entities, which end up in their own table columns.

Entities such as `ProductEntity` can be added to several of the new Ecommerce events, e.g. `ProductViewEvent` or `AddToCartEvent`. Reusing the same entities makes modeling easier and keeps the code cleaner.

## Using the dbt data model

Since the structure of the events is so different, migrating to the new ecommerce events is a breaking change for any relevant data models.

We provide a [dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) that creates a set of derived tables including carts, transactions, and products from the raw ecommerce event data.

---

# Send user identifiers to Kantar Focal Meter
> Integrate mobile trackers with Kantar Focal Meter to send user identifiers for audience measurement.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/plugins/focal-meter/

The mobile trackers provide integration with [Focal Meter by Kantar](https://www.virtualmeter.co.uk/focalmeter). Focal Meter is a box that connects directly to the broadband router and collects viewing information for the devices on your network.

This integration enables measuring the audience of content through the Focal Meter router meter. The tracker has the ability to send the user identifier (`userId` present in the session context) to a [Kantar Focal Meter](https://www.virtualmeter.co.uk/focalmeter) endpoint.

To enable this feature, you can pass the `FocalMeterConfiguration` configuration with the URL of the Kantar endpoint. For example:

**iOS:**

```swift
let focalMeterConfig = FocalMeterConfiguration(kantarEndpoint: "https://thekantarendpoint.com")

let tracker = Snowplow.createTracker(namespace: namespace, network: networkConfig, configurations: [focalMeterConfig])
```

**Android (Kotlin):**

```kotlin
val focalMeterConfig = FocalMeterConfiguration("https://thekantarendpoint.com")

val tracker = Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig)
```

**Android (Java):**

```java
FocalMeterConfiguration focalMeterConfig = new FocalMeterConfiguration("https://thekantarendpoint.com");

Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig);
```

***

Once configured, the tracker will make a request to the endpoint whenever the user identifier changes or each time the app starts.

> **Note:** Session context needs to be enabled (default) in order for the integration to work.

> **Note:** The feature is available from version 5.6.0 of the iOS and Android trackers.

## Processing the user ID sent in requests to Kantar

By default, the plugin sends the user ID as a GET parameter in requests to Kantar without modifying it. In case you want to apply some transformation on the value, such as hashing, you can provide the `processUserId` callback when instantiating the plugin:

**iOS:**

```swift
let focalMeterConfig = FocalMeterConfiguration(kantarEndpoint: "https://thekantarendpoint.com") { userId in

  return hash(userId) // apply custom hashing function

}

let tracker = Snowplow.createTracker(namespace: namespace, network: networkConfig, configurations: [focalMeterConfig])
```

**Android (Kotlin):**

```kotlin
val focalMeterConfig = FocalMeterConfiguration("https://thekantarendpoint.com") { userId ->

  hash(userId) // apply custom hashing function

}

val tracker = Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig)
```

**Android (Java):**

```java
FocalMeterConfiguration focalMeterConfig = new FocalMeterConfiguration(

  "https://thekantarendpoint.com",

  userId -> hash(userId) // apply custom hashing function

);

Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig);
```

***

---

# Native mobile tracker plugins
> Extend mobile tracker functionality using plugins to enrich events, filter tracking, or inspect tracked data.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/plugins/

> **Note:** Tracker plugins are only available since version 5 of the mobile trackers.

The mobile trackers provide a plugin architecture that allows new functionality to be added to the trackers. There are several Snowplow maintained plugins, however you are also free to build your own or leverage community plugins too.

Plugins provide the ability to intercept tracked events in order to enrich them with additional entities, filter them or just to inspect them. To implement this, they contain three extension points:

1. `entities` callback which can return context entities to enrich events before they are tracked. The callback is passed the tracked event object to be enriched.
2. `filter` callback that returns true or false in order to decide whether to track a given event or not. The callback is passed the tracked event object.
3. `afterTrack` callback which can be used to inspect final tracked events. The passed event object contains all its properties as well as context entities.

## Creating a custom plugin

In order to build your own plugin, you can instantiate the `PluginConfiguration` class. It requires you to pass a unique identifier. There can only be a single plugin for a given identifier registered with the tracker.

**iOS:**

```swift
let plugin = PluginConfiguration(identifier: "myPlugin")
```

**Android (Kotlin):**

```kotlin
val plugin = PluginConfiguration("myPlugin")
```

**Android (Java):**

```java
PluginConfiguration plugin = new PluginConfiguration("myPlugin");
```

***

### Adding context entities

To enrich events with additional context entities, you can make use of the `entities` callback. This function accepts an optional list of schemas of self-describing events to subscribe to. To filter primitive (not self-describing events) such as structured events, you can pass their event names in the schemas list (`se` for structured events). When no list of schemas is passed, the callback is called for all tracked events.

The following code will add a specific entity, with URI "iglu:xx" (not a valid URI, for example only), to all Structured and ScreenView events.

**iOS:**

```swift
plugin.entities(schemas: [

    // The list of schemas to call the closure for is optional. If not passed, the callback is called for all events.

    "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events

    "se" // structured events

]) { event in

    return [

        SelfDescribingJson(schema: "iglu:xx", andData: ["yy": true])

    ]

}
```

**Android (Kotlin):**

```kotlin
plugin.entities(

    schemas = listOf(

        // The list of schemas to call the closure for is optional. If not passed, the callback is called for all events.

        "iglu:com.google.analytics.measurement-protocol/screen_view/jsonschema/1-0-0", // screen view events

        "se" // structured events

    )

) {

    listOf(SelfDescribingJson("iglu:xx", hashMapOf("yy" to true)))

}
```

**Android (Java):**

```java
plugin.entities(

        Arrays.asList(

                // The list of schemas to call the closure for is optional. If not passed, the callback is called for all events.

                "iglu:com.google.analytics.measurement-protocol/screen_view/jsonschema/1-0-0", // screen view events

                "se" // structured events

        ),

        event -> Collections.singletonList(new SelfDescribingJson("iglu:xx", Collections.singletonMap("yy", true)))

);
```

***

### Filtering events

The `filter` callback enables you to add custom logic that decides whether a given event should be tracked or not. This can for example enable you to intercept events automatically tracked by the tracker and skip some of them. The callback returns true in case the event should be tracked and false otherwise. It also accepts the same optional `schemas` array as the `entities` callback to only be applied to events with those schemas.

The following code will apply to all screen view events and only accept ones with the name "Home Screen", other screen view events will be discarded:

**iOS:**

```swift
plugin.filter(schemas: [

    "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events

]) { event in

    return event.payload["name"] as? String == "Home Screen"

}
```

**Android (Kotlin):**

```kotlin
plugin.filter(

    schemas = listOf(

        "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events

    )

) { event ->

    event.payload["name"] == "Home Screen"

}
```

**Android (Java):**

```java
plugin.filter(

        Collections.singletonList(

                "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events

        ),

        event -> event.getPayload().get("name") == "Home Screen"

);
```

***

### Inspecting events after they are tracked

To inspect events after they are tracked, you can make use of the `afterTrack` callback. It also accepts the same optional `schemas` array as the `entities` and `filter` callbacks to filter events.

**iOS:**

```swift
plugin.afterTrack { event in

    print("Tracked event with \(event.entities.count) entities")

}
```

**Android (Kotlin):**

```kotlin
plugin.afterTrack {

    println("Tracked event with ${it.entities.count()} entities")

}
```

**Android (Java):**

```java
plugin.afterTrack(

        null,

        event -> System.out.printf("Tracked event with %d entities%n", event.getEntities().size())

);
```

***

## Registering a plugin in the tracker

Once you have a plugin configuration, you can register it when creating a new tracker:

**iOS:**

```swift
// the plugin is supplied to the tracker as a configuration

let tracker = Snowplow.createTracker(namespace: "namespace",

                                     network: networkConfig,

                                     configurations: [plugin])
```

**Android (Kotlin):**

```kotlin
// the plugin is supplied to the tracker as a configuration

val tracker = Snowplow.createTracker(

    applicationContext,

    "namespace",

    networkConfiguration,

    plugin

)
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.createTracker(

        getApplicationContext(),

        "namespace",

        networkConfiguration,

        plugin

);
```

***

You will be able to list the registered plugin identifiers:

**iOS:**

```swift
// one can inspect the enabled plugins and get the list of their identifiers

let pluginIdentifiers = tracker?.plugins.identifiers
```

**Android (Kotlin):**

```kotlin
// one can inspect the enabled plugins and get the list of their identifiers

val pluginIdentifiers = tracker.plugins.identifiers
```

**Android (Java):**

```java
// one can inspect the enabled plugins and get the list of their identifiers

List<String> identifiers = tracker.getPlugins().getIdentifiers();
```

***

You can also add new plugins to an already instantiated tracker object:

**iOS:**

```swift
tracker?.plugins.add(plugin: otherPlugin)
```

**Android (Kotlin):**

```kotlin
tracker.plugins.addPlugin(otherPlugin)
```

**Android (Java):**

```java
tracker.getPlugins().addPlugin(otherPlugin);
```

***

And remove a plugin from a tracker instance:

**iOS:**

```swift
tracker?.plugins.remove(identifier: "myPlugin")
```

**Android (Kotlin):**

```kotlin
tracker.plugins.removePlugin("myPlugin")
```

**Android (Java):**

```java
tracker.getPlugins().removePlugin("myPlugin");
```

***

---

# Update the native mobile trackers configuration remotely
> Configure mobile trackers to download and apply configuration updates remotely without requiring app resubmission.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/remote-configuration/

Remote configuration allows for the configuration of the tracker without distributing an app update. When remote configuration is not used, changing the tracking configuration can require the resubmission of the whole app bundle.

With remote configuration, the tracker can download a configuration file and automatically reconfigure itself based on the downloaded file.

A tracker that will be remotely configured is created slightly differently from a locally-configured tracker, which is created using `Snowplow.createTracker()`. Instead, `Snowplow.setup()` is used. This example code will create a tracker with namespace "defaultNamespace" that can be configured remotely. Access the tracker using `Snowplow.defaultTracker` or `Snowplow.getTracker("defaultNamespace")`.

**iOS:**

1. In your application delegate `AppDelegate.swift` add `import SnowplowTracker`.

2. In the `application(_:didFinishLaunchingWithOptions:)` method, set up the SDK as follows:

```swift
// Indicate the URL where to download the config file

let remoteConfig = RemoteConfiguration(

  endpoint: "https://remote-config.com",

  method: .get

)



// Default configuration used when the remote config is not accessible

// Network, tracker, subject and sessionConfigurations can be provided

let defaultNetworkConfig = NetworkConfiguration(endpoint: "https://snowplow-collector-url.com", method: .post)

let bundle = ConfigurationBundle(namespace: "defaultNamespace", networkConfiguration: defaultNetworkConfig)

bundle.trackerConfiguration = TrackerConfiguration(appId: "app-id")

  .screenViewAutotracking(true)

let defaultConfig = [bundle]



// Optional callback for when the tracker reconfigures itself passing a list of active namespaces and the state of the configuration describing where it was fetched from

let successCallback: ([String]?) -> Void = { namespaces, state in

  // This callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.

}



// Set up tracker with remote configuration

Snowplow.setup(

  remoteConfiguration: remoteConfig,

  defaultConfiguration: defaultConfig,

  defaultConfigurationVersion: 1,

  onSuccess: successCallback

)
```

**Android (Kotlin):**

```kotlin
// Indicate the URL where to download the config file

val remoteConfig = RemoteConfiguration("https://remote-config.com", HttpMethod.GET)



// Default configuration used in case the remote config is not accessible

// Network, tracker, subject and sessionConfigurations can be provided

val defaultNetworkConfig = NetworkConfiguration("https://snowplow-collector-url.com", HttpMethod.POST)

val defaultConfig: List<ConfigurationBundle> =

    listOf(ConfigurationBundle("defaultNamespace", defaultNetworkConfig))



// Set up tracker with remote configuration

Snowplow.setup(

  context = applicationContext,

  remoteConfiguration = remoteConfig,

  defaultBundles = defaultConfig,

  defaultBundleVersion = 1

) {

  // This optional callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.

  configurationPair: Pair<List<String>, ConfigurationState?>? ->

    val namespaces = configurationPair?.first

    val configurationState =

        configurationPair?.second // either cached, default or fetched from the remote endpoint

}
```

**Android (Java):**

```java
// Indicate the URL where to download the config file

RemoteConfiguration remoteConfig = new RemoteConfiguration("https://remote-config.com", HttpMethod.get);



// Default configuration used in case the remote config is not accessible

// Network, tracker, subject and sessionConfigurations can be provided

NetworkConfiguration defaultNetworkConfig = new NetworkConfiguration("https://snowplow-collector-url.com", HttpMethod.post);

List<ConfigurationBundle> defaultConfig = Lists.listOf(new ConfigurationBundle("defaultNamespace", defaultNetworkConfig));



// Set up tracker with remote configuration

Snowplow.setup(getApplicationContext(),

  remoteConfig,

  defaultConfig,

  1, // version of the default configuration

  configurationPair -> {

  // This optional callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.

  List<String> namespaces = configurationPair.first;

  ConfigurationState configurationState = configurationPair.second; // either cached, default or fetched from the remote endpoint

});
```

***

On app startup, the Snowplow tracker initializer will attempt to download the remote configuration file. Meanwhile, it will initialize the tracker instance (or multiple tracker instances) based on the last cached configuration. The cached configuration is the last configuration downloaded remotely. If it's not available, the tracker initializer will spin up the default configuration passed as a parameter.

Every time the initializer successfully initializes the tracker instances it calls a callback passing the list of activated namespaces and the state of the configuration which represents the source where the configuration was retrieved from (using default values, cache, or fetched from the remote endpoint). The callback can be used for last minute settings at runtime once the tracker has been instanced.

## Default configuration

The default configuration will be used right after calling the `setup()` method in case there is no cached configuration (one that was fetched from the remote endpoint and stored in a cache file).

The `defaultConfiguration` parameter takes a list of `ConfigurationBundle` objects. Each one represents a tracker instance and it's own configuration. Each tracker instance is identified by a `namespace` which is a parameter required in each `ConfigurationBundle`.

The optional `defaultConfigurationVersion` parameter specifies the version of the default configuration. If the fetched remote configuration has version lower or equal to the default configuration, it will not be used.

## On success callback

The optional `onSuccess` callback is called for each successful configuration.

For example, if the tracker has a cached configuration and it's downloading a new configuration, first the `onSuccess` callback will be called for the cached configuration and then it will be called when the downloaded configuration is applied. When it happens the cached configuration is substituted with the downloaded one.

## The remote configuration file

The URL where the configuration file is hosted must be provided in the `RemoteConfiguration` object passed to the `setup()` method. There aren't restrictions about where to host the file but possible solutions may be S3 with Cloudfront or a Google Cloud Storage bucket.

The configuration file is simply a JSON file compliant with the [Remote Config JSONSchema](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/remote_config/jsonschema/1-0-0).

An example of remote config specification:

```json
{

  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.mobile/remote_config/jsonschema/1-0-0",

  "configurationVersion": 1,

  "configurationBundle": [

    {

      "namespace": "testTracker1",

      "networkConfiguration": {

        "endpoint": "http://<collector.url>",

        "method": "get"

      },

      "trackerConfiguration": {

        "appId": "app-identifier",

        "devicePlatform": "mob",

        "logLevel": "off",

        "sessionContext": false,

        "applicationContext": false,

        "platformContext": false,

        "geoLocationContext": false,

        "screenContext": false,

        "screenViewAutotracking": false,

        "lifecycleAutotracking": false,

        "installAutotracking": false,

        "exceptionAutotracking": false,

        "diagnosticAutotracking": false

      },

      "subjectConfiguration": {

        "userId": "example",

        "networkUserId": "example",

        "domainUserId": "example",

        "useragent": "example",

        "ipAddress": "example",

        "timezone": "example",

        "language": "example"

      },

      "sessionConfiguration": {

        "backgroundTimeout": 1800,

        "foregroundTimeout": 1800

      }

    }

  ]

}
```

The required fields are:

- `$schema`: It specifies the format of the configuration file and it's used by the tracker to check if the format is compatible with that version of the tracker.

- `configurationVersion`: It's an incremental version number that identifies the current configuration. The tracker compares this value with the configurationVersion of the cached configuration to decide whether to update or not the tracker configuration.

  > **Note:** The version MUST be increased on each update.

- `configurationBundle`: This is a list of configurations for the various tracker instances the developer wants to activate in the app. Usually there is a unique tracker instance for the app so the configurationBundle will likely be an array of a single object like in the example above.

The elements of the `configurationBundle` list are just a subset of the common configuration settings described in the "Introduction" section. At the moment the configuration objects configurable remotely are:

- `networkConfiguration`
- `trackerConfiguration`
- `subjectConfiguration`
- `sessionConfiguration`

Note: The `networkConfiguration` is fundamental in order to set the collector endpoint, the URL where the tracked events will be sent. If the ConfigurationBundle has a `namespace` but not the `networkConfiguration`, the tracker initializer will remove the tracker instance with the corresponding namespace.

## Refresh the configuration

By default the tracker will attempt to download the configuration file at the start up of the app. To check for configuration updates more often, for example at runtime or when the app comes back from background state, it's possible to use the `refresh()` method.

**iOS:**

```swift
let successCallback: ([String]?) -> Void = { namespaces, state in

    // This callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.

}



Snowplow.refresh(onSuccess: successCallback)
```

**Android (Kotlin):**

```kotlin
Snowplow.refresh(

    applicationContext

) { configurationPair: Pair<List<String>, ConfigurationState>? ->

    val namespaces = configurationPair?.first

    val configurationState = configurationPair?.second

}
```

**Android (Java):**

```java
Snowplow.refresh(getApplicationContext(), configurationPair -> {

  List<String> namespaces = configurationPair.first;

  ConfigurationState configurationState = configurationPair.second;



  // This callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.

});
```

***

The method needs only the callback parameter, without any remote configuration url or default configuration, because this is intended just for the configuration updates, not for the initial setup. The tracker will be configured only if a new configuration (with a higher `configurationVersion` value) is available at the url indicated in the `RemoteConfiguration` passed earlier at start up of the app in the `setup` method call.

## Policy for runtime settings and remote configuration updates

When the tracker initializer updates the tracker configuration, it would reset all the previous configuration with the new settings. Obviously, this can cause issues in case some runtime configuration has been applied meanwhile. To avoid this the tracker keeps track of runtime changes in the configuration settings and when a new remote configuration is downloaded and applied, it doesn't override the settings already changed at runtime.

A clear example is a runtime change on `userId` on `SubjectController`:

**iOS:**

```swift
// Load configuration with userId = nil

Snowplow.setup(remoteConfiguration: remoteConfig, defaultConfiguration: defaultConfig, onSuccess: successCallback)



// userId is set to nil



// ...later...



tracker.subject.userId = "my-runtime-updated-userId"



// userId is is set to "my-runtime-updated-userId"



// ...later...



// Later refreshing the configuration with userId = nil

Snowplow.refresh(onSuccess: successCallback)



// userId is still set to "my-runtime-updated-userId" because it was set at runtime
```

**Android (Kotlin):**

```kotlin
// Load configuration with userId = nil

Snowplow.setup(applicationContext, remoteConfig, defaultConfig, successCallback)



// userId is set to nil



// ...later...



tracker.subject.userId = "my-runtime-updated-userId"



// userId is is set to "my-runtime-updated-userId"



// ...later...



// Later refreshing the configuration with userId = nil

Snowplow.refresh(applicationContext, successCallback)



// userId is still set to "my-runtime-updated-userId" because it was set at runtime
```

**Android (Java):**

```java
// Load configuration with userId = nil

Snowplow.setup(getApplicationContext(), remoteConfig, defaultConfig, successCallback);



// userId is set to nil



// ...later...



tracker.getSubject().userId = "my-runtime-updated-userId"



// userId is is set to "my-runtime-updated-userId"



// ...later...



// Later refreshing the configuration with userId = nil

Snowplow.refresh(getApplicationContext(), successCallback);



// userId is still set to "my-runtime-updated-userId" because it was set at runtime
```

***

---

# Track ecommerce events with the native mobile trackers
> Track comprehensive ecommerce interactions including product views, cart actions, checkouts, and transactions in mobile apps.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/

> **Note:** Snowplow ecommerce tracking was added in version 5.4.0. With the addition of these new ecommerce events and entities, we have deprecated the old `EcommerceTransaction` and `EcommerceTransactionItem` events. [Migration guide](/docs/sources/mobile-trackers/migration-guides/migration-guide-to-new-ecommerce/).

The Snowplow ecommerce tracking APIs enable you to track events from your ecommerce store on the [web](/docs/sources/web-trackers/tracking-events/ecommerce/) as well as mobile apps.

All ecommerce events must be manually tracked; there is no ecommerce auto-tracking.

## Ecommerce events

Ecommerce events are tracked like normal Snowplow events. For example, tracking an ecommerce `ProductViewEvent` event:

**iOS:**

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")



let product = ProductEntity(

  id: "productId",

  category: "category",

  currency: "GBP",

  price: 100

)

let event = ProductViewEvent(product: product)

let eventId = tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val tracker = Snowplow.createTracker(

    context = applicationContext, // Android context

    namespace = "appTracker",

    endpoint = "https://snowplow-collector-url.com" // Event collector URL

)



val event = ProductViewEvent(ProductEntity(

  id = "productId",

  category = "category",

  currency = "GBP",

  price = 100

  )

)

tracker.track(event)
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.createTracker(

      getApplicationContext(), // Android context

      "appTracker", // namespace

      "https://snowplow-collector-url.com" // Event collector URL

);



ProductViewEvent event = new ProductViewEvent(new ProductEntity(

  "productId", // id

  "category",  // category

  "GBP", // currency

  100 // price

  )

);

tracker.track(event);
```

***

Add or update properties using setters. For example, adding data to a PromotionEntity:

**iOS:**

```swift
let promotion = PromotionEntity(id: "promoId")

promotion.name = "bogof"

promotion.type = "popup"
```

**Android (Kotlin):**

```kotlin
val promotion = PromotionEntity(id = "promoId")

promotion.name = "bogof"

promotion.type = "popup"
```

**Android (Java):**

```java
PromotionEvent promotion = new PromotionEntity("promoId");

promotion.setName("bogof");

promotion.setType("popup");
```

***

This table lists all the ecommerce events.

| `Event`                 | Used for                                                                                                                                       |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `ProductViewEvent`      | Tracking a visit to a product detail screen. Also known as product detail view.                                                                |
| `AddToCartEvent`        | Track an addition to cart.                                                                                                                     |
| `RemoveFromCartEvent`   | Track a removal from cart.                                                                                                                     |
| `ProductListViewEvent`  | Track an impression of a product list. The list could be a search results page, recommended products, upsells etc.                             |
| `ProductListClickEvent` | Track the click/selection of a product from a product list.                                                                                    |
| `PromotionViewEvent`    | Track an impression for an internal promotion banner or slider or any other type of content that showcases internal products/categories.       |
| `PromotionClickEvent`   | Track the click/selection of an internal promotion.                                                                                            |
| `CheckoutStepEvent`     | Track a checkout step completion in the checkout process together with common step attributes for user choices throughout the checkout funnel. |
| `TransactionEvent`      | Track a transaction/purchase completion.                                                                                                       |
| `TransactionErrorEvent` | Track a failed transaction.                                                                                                                    |
| `RefundEvent`           | Track a transaction partial or complete refund.                                                                                                |

Each ecommerce event is a [self-describing](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) event using a single schema, `iglu:com.snowplowanalytics.snowplow.ecommerce/snowplow_ecommerce_action/jsonschema/1-0-2`.

**Ecommerce action event properties**

| Request Key | Required | Type/Format | Description                                                                   |
| ----------- | -------- | ----------- | ----------------------------------------------------------------------------- |
| type        | Y        | string      | Specific ecommerce event type (automatically tracked).                        |
| name        | N        | string      | For `ProductListView` and `ProductListClick` events: human-readable list name |

The events are distinguished by their `type` property, which is different for each `Event` class tracked. Aside from the optional list `name` in the `ProductListViewEvent` and `ProductListClickEvent` events, all tracked ecommerce properties are tracked as entities.

> **Note:** Check out the API docs ([Android](https://snowplow.github.io/snowplow-android-tracker/), [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/snowplow/)) for the full details of each Event and Entity.

### ProductView

Tracking a visit to a details screen for a specific product.

**iOS:**

```swift
let product = ProductEntity(

  id: "plow2",

  category: "snow.clearance.ploughs.large",

  currency: "NOK",

  price: 5000

)

let event = ProductViewEvent(product: product)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(

  id = "plow2",

  category = "snow.clearance.ploughs.large",

  currency = "NOK",

  price = 5000

)

val event = ProductViewEvent(product)



tracker.track(event)
```

**Android (Java):**

```java
ProductViewEvent event = new ProductViewEvent(new ProductEntity(

    "plow2", // id

    "snow.clearance.ploughs.large",  // category

    "NOK", // currency

    5000 // price

  )

);



tracker.track(event);
```

***

### AddToCart

Tracking one or more products being added to a cart.

**iOS:**

```swift
let product = ProductEntity(

  id: "productId",

  category: "clothes/shirts",

  currency: "EUR",

  price: 100.50

)

let cart = CartEntity(totalValue: 200, currency: "EUR")

let event = AddToCartEvent(products: [product], cart: cart)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(

  id = "productId",

  category = "clothes/shirts",

  currency = "EUR",

  price = 100.50

)

val cart = CartEntity(totalValue = 200, currency = "EUR")

val event = AddToCartEvent(listOf(product), cart)



tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(

  "productId", // id

  "clothes/shirts", // category

  "EUR", // currency

  100.50 // price

);

List<Product> products = new ArrayList<>();

products.add(product);



CartEntity cart = new CartEntity(

  200, // totalValue

  "EUR" // currency

);

AddToCartEvent event = new AddToCartEvent(products, cart);



tracker.track(event);
```

***

### RemoveFromCart

Tracking one or more products being removed from a cart.

**iOS:**

```swift
let product = ProductEntity(id: "productId", category: "clothes/shirts", currency: "EUR", price: 100.50)

let cart = CartEntity(totalValue: 200, currency: "EUR")

let event = RemoveFromCartEvent(products: [product], cart: cart)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(

  id = "productId",

  category = "clothes/shirts",

  currency = "EUR",

  price = 100.50

)

val cart = CartEntity(totalValue = 200, currency = "EUR")

val event = RemoveFromCartEvent(listOf(product), cart)



tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(

  "productId", // id

  "clothes/shirts", // category

  "EUR", // currency

  100.50 // price

);

List<Product> products = new ArrayList<>();

products.add(product);



CartEntity cart = new CartEntity(

  200, // totalValue

  "EUR" // currency

);

RemoveFromCartEvent event = new RemoveFromCartEvent(products, cart);



tracker.track(event);
```

***

### ProductListView

Track an impression of a list of products. You can optionally provide a name for the list.

**iOS:**

```swift
let product = ProductEntity(id: "productId", category: "software", currency: "USD", price: 99.99)

let event = ProductListViewEvent(products: [product], name: "snowplowProducts")



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(

  id = "productId",

  category = "software",

  currency = "USD",

  price = 99.99

)

val event = ProductListViewEvent(listOf(product), name = "snowplowProducts")



tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(

  "productId", // id

  "software", // category

  "USD", // currency

  99.99 // price

);

List<Product> products = new ArrayList<>();

products.add(product);

ProductListViewEvent event = new ProductListViewEvent(

  products, // products

  "snowplowProducts" // name

);



tracker.track(event);
```

***

### ProductListClick

Track a specific product being selected from a list. You can optionally provide a name for the list.

**iOS:**

```swift
let product = ProductEntity(id: "productId", category: "software", currency: "USD", price: 99.99)

let event = ProductListClickEvent(product: product, name: "snowplowProducts")



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(

  id = "productId",

  category = "software",

  currency = "USD",

  price = 99.99

)

val event = ProductListClickEvent(product, name = "snowplowProducts")



tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(

  "productId", // id

  "software", // category

  "USD", // currency

  99.99 // price

);

ProductListClickEvent event = new ProductListClickEvent(

  product, // product

  "snowplowProducts" // name

);



tracker.track(event);
```

***

### CheckoutStep

Track the completion of a step in the checkout funnel, along with common checkout properties such as payment method or coupon code. See the API docs for the full details ([Android](https://snowplow.github.io/snowplow-android-tracker/), [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/snowplow/)).

**iOS:**

```swift
let event = CheckoutStepEvent(step: 3, deliveryMethod: "next_day")



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = CheckoutStepEvent(step = 3, deliveryMethod = "next_day")



tracker.track(event)
```

**Android (Java):**

```java
CheckoutStepEvent event = new CheckoutStepEvent(

  3, // step

  null, // shippingPostcode

  null, // billingPostcode

  null, // shippingFullAddress

  null, // billingFullAddress

  "next_day" // deliveryMethod

);



tracker.track(event);
```

***

### Transaction

Track the completion of a purchase or transaction, along with common transaction properties such as shipping cost or discount amount.

**iOS:**

```swift
let transaction = TransactionEntity(

  transactionId: "id-123",

  revenue: 50000,

  currency: "JPY",

  paymentMethod: "debit",

  totalQuantity: 2

)

let event = TransactionEvent(transaction)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val transaction = TransactionEntity(

  id = "id-123",

  revenue = 50000,

  currency = "JPY",

  paymentMethod = "debit",

  totalQuantity = 2

)

val event = TransactionEvent(transaction)



tracker.track(event)
```

**Android (Java):**

```java
TransactionEntity transaction = new TransactionEntity(

  "id-123", // id

  50000, // revenue

  "JPY", // currency

  "debit", // paymentMethod

  2 // totalQuantity

);

TransactionEvent event = new TransactionEvent(transaction);



tracker.track(event);
```

***

### TransactionError

Track an error occurring during a transaction.

**iOS:**

```swift
let transaction = TransactionEntity(

  transactionId: "id-123",

  revenue: 50000,

  currency: "JPY",

  paymentMethod: "debit",

  totalQuantity: 2

)

let event = TransactionErrorEvent(

  transaction: transaction,

  errorCode: "E05",

  errorDescription: "connection_failure"

)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val transaction = TransactionEntity(

  id = "id-123",

  revenue = 50000,

  currency = "JPY",

  paymentMethod = "debit",

  totalQuantity = 2

)

val event = TransactionErrorEvent(

    transaction = transaction,

    errorCode = "E05",

    errorDescription = "connection_failure"

)



tracker.track(event)
```

**Android (Java):**

```java
TransactionEntity transaction = new TransactionEntity(

  "id-123", // id

  50000, // revenue

  "JPY", // currency

  "debit", // paymentMethod

  2 // totalQuantity

);

TransactionErrorEvent event = new TransactionErrorEvent(

    transaction, // transaction

    "E05", // errorCode

    null, // errorShortcode

    "connection_failure" // errorDescription

);



tracker.track(event);
```

***

### Refund

Track a refund being requested for part of, or the entirety of, a transaction.

**iOS:**

```swift
let event = RefundEvent(

  transactionId: "id-123", // use the transaction ID from the original Transaction event

  refundAmount: 20000,

  currency: "JPY"

)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = Refund(

    id = "id-123", // use the transaction ID from the original Transaction event

    refundAmount = 20000,

    currency = "JPY"

)



tracker.track(event)
```

**Android (Java):**

```java
RefundEvent event = new RefundEvent(

    "id-123", // id; use the transaction ID from the original Transaction event

    20000, // refundAmount

    "JPY" // currency

);



tracker.track(event);
```

***

### PromotionView

Track an impression for any kind of internal promotion.

**iOS:**

```swift
let promotion = PromotionEntity(id: "promoId")

let event = PromotionViewEvent(promotion: promotion)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val promotion = PromotionEntity(id = "promoId")

val event = PromotionViewEvent(promotion)



tracker.track(event)
```

**Android (Java):**

```java
PromotionEntity promotion = new PromotionEntity("promoId");

PromotionViewEvent event = new PromotionViewEvent(promotion);



tracker.track(event);
```

***

### PromotionClick

Track an internal promotion being selected.

**iOS:**

```swift
let promotion = PromotionEntity(id: "promoId")

let event = PromotionViewEvent(promotion: promotion)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val promotion = PromotionEntity(id = "promoId")

val event = PromotionClick(promotion)



tracker.track(event)
```

**Android (Java):**

```java
PromotionEntity promotion = new PromotionEntity("promoId");

PromotionClickEvent event = new PromotionClickEvent(promotion);



tracker.track(event);
```

***

## Ecommerce entities

### Product entity

The product entity is used in several ecommerce events. It contains information about the product the user is interacting with.

**Product entity properties**

| Request Key     | Required | Type/Format | Description                                |
| --------------- | -------- | ----------- | ------------------------------------------ |
| id              | Y        | string      | The SKU or product ID.                     |
| category        | Y        | string      | Category the product belongs to.           |
| currency        | Y        | string      | Currency in which the product is priced.   |
| price           | Y        | number      | Current price.                             |
| listPrice       | N        | boolean     | Recommended or list price.                 |
| name            | N        | string      | Product name or title.                     |
| quantity        | N        | integer     | Quantity of the product (for cart events). |
| size            | N        | string      | Product size.                              |
| variant         | N        | string      | Product variant.                           |
| brand           | N        | string      | Product brand.                             |
| inventoryStatus | N        | string      | Inventory status, e.g. in stock.           |
| position        | N        | integer     | Position in a list of products.            |
| creativeId      | N        | string      | Identifier for the creative shown.         |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/product/jsonschema/1-0-0`.

### Cart entity

The cart entity is used for `AddToCart` and `RemoveFromCart` events.

**Cart entity properties**

| Request Key | Required | Type/Format | Description                         |
| ----------- | -------- | ----------- | ----------------------------------- |
| totalValue  | Y        | number      | Total cart value after this action. |
| currency    | Y        | string      | Currency used for the cart.         |
| cartId      | N        | string      | Unique cart identifier.             |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/cart/jsonschema/1-0-0`.

### Checkout step entity

The checkout step entity is used for `CheckoutStep` events.

**Checkout step entity properties**

| Request Key         | Required | Type/Format | Description                                                       |
| ------------------- | -------- | ----------- | ----------------------------------------------------------------- |
| step                | Y        | integer     | Checkout step index.                                              |
| shippingPostcode    | N        | string      | Shipping address postcode.                                        |
| billingPostcode     | N        | string      | Billing address postcode.                                         |
| shippingFullAddress | N        | string      | Shipping address.                                                 |
| billingFullAddress  | N        | string      | Billing address.                                                  |
| deliveryProvider    | N        | string      | Delivery provider e.g. DHL, Royal Mail.                           |
| deliveryMethod      | N        | string      | How the customer will get the product.                            |
| couponCode          | N        | string      | An applied coupon.                                                |
| accountType         | N        | string      | Customer account type, e.g. guest                                 |
| paymentMethod       | N        | string      | Payment method selected.                                          |
| proofOfPayment      | N        | string      | e.g. invoice, receipt                                             |
| marketingOptIn      | N        | boolean     | Whether the user email address is opted into marketing campaigns. |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/checkout_step/jsonschema/1-0-0`.

### Transaction entity

The transaction entity is used for `Transaction` and `TransactionError` events.

**Transaction entity properties**

| Request Key    | Required | Type/Format | Description                                |
| -------------- | -------- | ----------- | ------------------------------------------ |
| transactionId  | Y        | string      | Unique transaction identifier.             |
| revenue        | Y        | number      | Value of transaction.                      |
| currency       | Y        | string      | Currency used in transaction.              |
| paymentMethod  | Y        | string      | Payment method used.                       |
| totalQuantity  | Y        | integer     | Number of items in the transaction.        |
| tax            | N        | number      | Tax value in transaction.                  |
| shipping       | N        | number      | Shipping costs in transaction.             |
| discountCode   | N        | string      | An applied discount code.                  |
| discountAmount | N        | string      | Amount discounted from transaction.        |
| creditOrder    | N        | boolean     | Whether the transaction is a credit order. |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/transaction/jsonschema/1-0-0`.

### Transaction error entity

The transaction error entity is used for `TransactionError` events.

**Transaction error entity properties**

| Request Key      | Required | Type/Format | Description                    |
| ---------------- | -------- | ----------- | ------------------------------ |
| errorCode        | N        | string      | Error-identifying code.        |
| errorShortcode   | N        | string      | Shortcode for the error.       |
| errorDescription | N        | string      | Longer description.            |
| errorType        | N        | string enum | Is the error "hard" or "soft". |
| resolution       | N        | string      | The chosen error resolution.   |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/transaction_error/jsonschema/1-0-0`.

### Refund entity

The refund entity is used for `Refund` events.

**Refund entity properties**

| Request Key   | Required | Type/Format | Description                                   |
| ------------- | -------- | ----------- | --------------------------------------------- |
| transactionId | Y        | string      | The identifier from the original transaction. |
| refundAmount  | Y        | number      | Amount to be refunded.                        |
| currency      | Y        | string      | Currency used in transaction.                 |
| refundReason  | N        | string      | Reason for the refund.                        |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/refund/jsonschema/1-0-0`.

### Promotion entity

The promotion entity is used for `PromotionView` and `PromotionClick` events.

**Promotion entity properties**

| Request Key | Required | Type/Format     | Description                                        |
| ----------- | -------- | --------------- | -------------------------------------------------- |
| id          | Y        | string          | The unique promotion identifier.                   |
| name        | N        | string          | Promotion name.                                    |
| productIds  | N        | list of strings | IDs of products in the promotion.                  |
| position    | N        | integer         | Index of the promotion in a list such as a banner. |
| creativeId  | N        | string          | Identifier for the promotion creative.             |
| type        | N        | string          | Type of promotion.                                 |
| slot        | N        | string          | Where the promotion was displayed.                 |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/promotion/jsonschema/1-0-0`.

## Global ecommerce entities Screen/Page and User

Use these APIs to add ecommerce context information to every subsequent event tracked.

> **Note:** These entities will be added to **all** events, not just ecommerce ones. This is for consistency with entities in the JavaScript tracker.

### Ecommerce Screen (Page) entity

> **Note:** The `setEcommerceScreen` method adds a `Page` (rather than `Screen`) entity to all events, for consistency with web tracking.

The Ecommerce Screen/Page entity helps in grouping insights by screen/page type, e.g. Product description, Product list, Home screen.

To set a Screen/Page entity you can use the `setEcommerceScreen` method:

**iOS:**

```swift
let entity = EcommerceScreenEntity(type: "demo_app_screen")

Snowplow.defaultTracker()?.ecommerce.setEcommerceScreen(entity)



// setting EcommerceScreen again will replace the original entity

let newEntity = EcommerceScreenEntity(type: "product_list", language: "EN-GB", locale: "UK")

Snowplow.defaultTracker()?.ecommerce.setEcommerceScreen(newEntity)



// remove the saved properties and stop the Page entity being added

Snowplow.defaultTracker()?.ecommerce.removeEcommerceScreen()
```

**Android (Kotlin):**

```kotlin
val entity = EcommerceScreenEntity(type = "demo_app_screen")

Snowplow.defaultTracker?.ecommerce.setEcommerceScreen(entity)



// setting EcommerceScreen again will replace the original entity

val newEntity = EcommerceScreenEntity(type = "product_list", language = "EN-GB", locale = "UK")

Snowplow.defaultTracker?.ecommerce.setEcommerceScreen(newEntity)



// remove the saved properties and stop the Page entity being added

Snowplow.defaultTracker?.ecommerce.removeEcommerceScreen()
```

**Android (Java):**

```java
EcommerceScreenEntity entity = new EcommerceScreenEntity(

  "demo_app_screen" // type

)

Snowplow.getDefaultTracker().getEcommerce().setEcommerceScreen(entity);



// setting EcommerceScreen again will replace the original entity

EcommerceScreenEntity newEntity = new EcommerceScreenEntity(

  "product_list", // type

  "EN-GB", // language

  "UK" // locale

)

Snowplow.getDefaultTracker().getEcommerce().setEcommerceScreen(newEntity);



// remove the saved properties and stop the Page entity being added

Snowplow.getDefaultTracker().getEcommerce().removeEcommerceScreen();
```

***

**Screen/Page entity properties**

| Request Key | Required | Type/Format | Description                   |
| ----------- | -------- | ----------- | ----------------------------- |
| type        | Y        | string      | Type of screen.               |
| language    | N        | string      | Language used for the screen. |
| locale      | N        | string      | Locale version.               |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/page/jsonschema/1-0-0`.

### Ecommerce User entity

The Ecommerce User entity helps in modeling guest/non-guest account interactions.

To set an Ecommerce User entity you can use the `setEcommerceUser` method with the following attributes:

**iOS:**

```swift
let entity = EcommerceUserEntity(id: "userId12345", isGuest: true)

Snowplow.defaultTracker()?.ecommerce.setEcommerceUser(entity)



// setting EcommerceUser again will replace the original entity

let newEntity = EcommerceUserEntity(id: "userId67890", isGuest: false, email: "email@email.com")

Snowplow.defaultTracker()?.ecommerce.setEcommerceUser(newEntity)



// remove the saved properties and stop the User entity being added

Snowplow.defaultTracker()?.ecommerce.removeEcommerceUser()
```

**Android (Kotlin):**

```kotlin
val entity = EcommerceUserEntity(id = "userId12345", isGuest = true)

Snowplow.defaultTracker?.ecommerce.setEcommerceUser(entity)



// setting EcommerceUser again will replace the original entity

val newEntity = EcommerceUserEntity(id = "userId67890", isGuest = false, email = "email@email.com")

Snowplow.defaultTracker?.ecommerce.setEcommerceUser(newEntity)



// remove the saved properties and stop the User entity being added

Snowplow.defaultTracker?.ecommerce.removeEcommerceUser()
```

**Android (Java):**

```java
EcommerceUserEntity entity = new EcommerceUserEntity(

  "userId12345", // id

  true // isGuest

)

Snowplow.getDefaultTracker().getEcommerce().setEcommerceUser(entity);



// setting ScreenType again will replace the original entity

EcommerceUserEntity newEntity = new EcommerceUserEntity(

  "userId67890", // id

  false, // isGuest

  "email@email.com" // email

)

Snowplow.getDefaultTracker().getEcommerce().setEcommerceUser(newEntity);



// remove the saved properties and stop the Page entity being added

Snowplow.getDefaultTracker().getEcommerce().removeEcommerceUser();
```

***

**User entity properties**

| Request Key | Required | Type/Format | Description                  |
| ----------- | -------- | ----------- | ---------------------------- |
| id          | Y        | string      | The unique user identifier.  |
| isGuest     | N        | boolean     | Whether the user is a guest. |
| email       | N        | string      | User email address.          |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/user/jsonschema/1-0-0`.

---

# Track exceptions with the native mobile trackers
> Automatically capture and track unhandled exceptions and crashes within mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/exception-tracking/

Exception tracking captures any unhandled exceptions within the application.

The exception tracking is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .exceptionAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .exceptionAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .exceptionAutotracking(true);
```

***

It allows the tracker to intercept critical exceptions in the app. Exceptions can crash the app so it's likely that the event will be sent after the restart of the app. Being a critical situation we can't be 100% sure that all the exception stacktraces are reliably stored for sending before the crash of the app.

---

# Track consent and GDPR with the native mobile trackers
> Add GDPR consent context entities to all tracked events for legal compliance and privacy management.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/gdpr-tracking/

You can add an [entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/gdpr/jsonschema/1-0-0) detailing the GDPR basis for processing to every event tracked. To configure this, provide a `GdprConfiguration` object when setting up a new tracker.

The only required argument is the `Basis` enum, which contains the six GDPR options. A document describing the legal basis for processing can also be provided (as a string), along with the document version and ID.

**iOS:**

```swift
let gdprConfig = GDPRConfiguration(

  basis: .consent, // basis for processing

  documentId: "id", // document ID

  documentVersion: "1.0", // document version

  documentDescription: "Legal justification" // document

)



let networkConfig = NetworkConfiguration(

  endpoint: "https://collector-url.com",

  method: .post

)



Snowplow.createTracker(

  namespace: "namespace",

  network: networkConfig,

  configurations: [gdprConfig]

)
```

**Android (Kotlin):**

```kotlin
val gdprConfiguration = GdprConfiguration(

    Basis.CONSENT, // basis for processing

    "someId", // document ID

    "0.1.0", // document version

    "Legal justification" // document

)



val networkConfiguration = NetworkConfiguration("https://collector-url.com")



createTracker(

    applicationContext,

    "namespace",

    networkConfiguration,

    gdprConfiguration

)
```

**Android (Java):**

```java
GdprConfiguration gdprConfiguration = new GdprConfiguration(

        Basis.CONSENT, // basis for processing

        "someId", // document ID

        "0.1.0", // document version

        "Legal justification" // document

);



NetworkConfiguration networkConfiguration = new NetworkConfiguration("https://collector-url.com");



Snowplow.createTracker(

        getApplicationContext(),

        "namespace",

        networkConfiguration,

        gdprConfiguration

);
```

***

---

# Track out-of-the-box events with the native mobile trackers
> Track pre-built event types including screen views, lifecycle events, sessions, and custom self-describing events in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/

To track an event, pass an event to the tracker instance.

For example, tracking a ScreenView:

**iOS:**

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")



let event = ScreenView(name: "screen name")

let eventId = tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val tracker = Snowplow.createTracker(

        applicationContext, // Android context

        "appTracker", // namespace

        "https://snowplow-collector-url.com" // Event collector URL

    )



val event = ScreenView("screen")

tracker.track(event)
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.createTracker(

      getApplicationContext(), // Android context

      "appTracker", // namespace

      "https://snowplow-collector-url.com" // Event collector URL

);



ScreenView event = new ScreenView("screen");

tracker.track(event);
```

***

The tracker makes it easy to track different kinds of data. We provide a range of `Event` classes for tracking out-of-the-box event types, as well as fully custom events.

Each event has an associated context, which is composed of entities. The tracker attaches entities to the events based on the configuration, but you can attach your own [custom entities](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) as well.

Every tracked event payload has a unique `event_id` UUID string set by the tracker, a set of timestamps, and other ubiquitous properties such as the `namespace`. The `event_id` is returned from the `tracker.track(event)` method. You can know more about how events and entities are structured [here](/docs/events/).

See the full configuration and parameter options for all these classes and methods in the API docs - [Android](https://snowplow.github.io/snowplow-android-tracker/index.html) and [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/).

## Auto-tracked events and entities

Automatically captured data are:

- [**Platform and Application Context Tracking**](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/): Captures contextual information about the device and the app.
- [**Session Tracking**](/docs/sources/mobile-trackers/tracking-events/session-tracking/): Captures the session which helps to keep track of the user activity in the app.
- [**App Lifecycle Tracking**](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/): Captures application lifecycle state changes (foreground/background transitions).
- [**Screen View and Engagement Tracking**](/docs/sources/mobile-trackers/tracking-events/screen-tracking/): Captures each time a new “screen” is loaded.
- [**Exception Tracking**](/docs/sources/mobile-trackers/tracking-events/exception-tracking/): Captures any unhandled exceptions within the application.
- [**Installation Tracking**](/docs/sources/mobile-trackers/tracking-events/installation-tracking/): Captures an install event which occurs the first time an application is opened.
- [**Immersive Space Tracking**](/docs/sources/mobile-trackers/tracking-events/visionos/): iOS only. Captures the visionOS immersive space in which events occur.

Autotracking can be enabled in the tracker configuration. In this example, some helpful automatic entities and all autotracking is enabled:

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", network: networkConfig) {

    TrackerConfiguration()

        .platformContext(true)

        .applicationContext(true)

        .lifecycleAutotracking(true)

        .sessionContext(true)

        .screenViewAutotracking(true)

        .screenEngagementAutotracking(true) // available from v6 of the tracker

        .screenContext(true)

        .exceptionAutotracking(true)

        .installAutotracking(true)

        .immersiveSpaceContext(true)

}
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .platformContext(true)

    .applicationContext(true)

    .lifecycleAutotracking(true)

    .sessionContext(true)

    .screenViewAutotracking(true)

    .screenEngagementAutotracking(true) // available from v6 of the tracker

    .screenContext(true)

    .exceptionAutotracking(true)

    .installAutotracking(true)



Snowplow.createTracker(

    applicationContext,

    namespace,

    networkConfiguration,

    trackerConfig

)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .platformContext(true)

    .applicationContext(true)

    .lifecycleAutotracking(true)

    .sessionContext(true)

    .screenViewAutotracking(true)

    .screenEngagementAutotracking(true) // available from v6 of the tracker

    .screenContext(true)

    .exceptionAutotracking(true)

    .installAutotracking(true);



Snowplow.createTracker(getApplicationContext(),

                namespace,

                networkConfiguration,

                trackerConfig);
```

***

You can also configure your tracker to automatically add [GDPR information](/docs/sources/mobile-trackers/tracking-events/gdpr-tracking/) to every event.

## Manually-tracked events

The tracker provides classes for tracking different types of events. The events are divided into two groups: canonical events and self-describing events.

### Creating a Timing event

Use the `Timing` events to track user timing events such as how long resources take to load.

The event schema is `iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0`.

**iOS:**

```swift
let event = Timing(category: "timing-category", variable: "timing-variable", timing: 5)

    .label("optional-label")



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = Timing("timing-category", "timing-variable", 5)

    .label("optional-label")



tracker.track(event)
```

**Android:**

```java
Timing event = new Timing("timing-category", "timing-variable", 5)

    .label("optional-label");



tracker.track(event);
```

***

### Creating a ScreenView event

Track the user viewing a screen within the application. This type of tracking is typically used when automatic [screen view tracking](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) is not suitable within your application.

The event schema is `iglu:com.snowplowanalytics.mobile/screen_view/jsonschema/1-0-0`.

**iOS:**

```swift
let event = ScreenView(name: "DemoScreenName", screenId: UUID())



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ScreenView("DemoScreenName")



tracker.track(event)
```

**Android (Java):**

```java
ScreenView event = new ScreenView("DemoScreenName");



tracker.track(event);
```

***

### Creating a Consent event

#### Consent Granted

Use the `ConsentGranted` event to track a user opting into data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied.

The event schema is `iglu:com.snowplowanalytics.snowplow/consent_granted/jsonschema/1-0-0`. The consent document schema is `iglu:com.snowplowanalytics.snowplow/consent_document/jsonschema/1-0-0`.

**iOS:**

```swift
let event = ConsentGranted(expiry: "2022-01-01T00:00:00Z", documentId: "1234abcd", version: "1.2")

    .name("document-name")

    .documentDescription("document-description")



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ConsentGranted("2022-01-01T00:00:00Z", "1234abcd", "1.2")

    .documentDescription("document-description")

    .documentName("document-name")



tracker.track(event)
```

**Android (Java):**

```java
ConsentGranted event = new ConsentGranted("2022-01-01T00:00:00Z", "1234abcd", "1.2")

        .documentDescription("document-description")

        .documentName("document-name");



tracker.track(event);
```

***

#### Consent Withdrawn

Use the `ConsentWithdrawn` event to track a user withdrawing consent for data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied. To specify that a user opts out of all data collection, `all` should be set to `true`.

The event schema is `iglu:com.snowplowanalytics.snowplow/consent_withdrawn/jsonschema/1-0-0`. The consent document schema is `iglu:com.snowplowanalytics.snowplow/consent_document/jsonschema/1-0-0`.

**iOS:**

```swift
let event = ConsentWithdrawn()

    .all(true)

    .documentId("1234abcd")

    .version("1.2")

    .name("document-name")

    .documentDescription("document-description")



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ConsentWithdrawn(false, "1234abcd", "1.2")

    .documentDescription("document-description")

    .documentName("document-name")



tracker.track(event)
```

**Android (Java):**

```java
ConsentWithdrawn event = new ConsentWithdrawn(false, "1234abcd", "1.2")

        .documentDescription("document-description")

        .documentName("document-name");



tracker.track(event);
```

***

### Tracking Push and Local Notifications

To track an event when a push (or local) notification is used, it is possible to use the `MessageNotification` event.

The event schema is `iglu:com.snowplowanalytics.mobile/message_notification/jsonschema/1-0-0`.

**iOS:**

```swift
let event = MessageNotification(title: "title", body: "body", trigger: .push)

    .notificationTimestamp("2020-12-31T15:59:60-08:00")

    .action("action")

    .bodyLocKey("loc key")

    .bodyLocArgs(["loc arg1", "loc arg2"])

    .sound("chime.mp3")

    .notificationCount(9)

    .category("category1")

    .attachments([

        MessageNotificationAttachment(identifier: "id", type: "type", url: "https://snowplow.io")

    ]);

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = MessageNotification("title", "body", MessageNotificationTrigger.push)

    .notificationTimestamp("2021-10-18T10:16:08.008Z")

    .category("category")

    .action("action")

    .bodyLocKey("loc key")

    .bodyLocArgs(listOf("loc arg1", "loc arg2"))

    .sound("chime.mp3")

    .notificationCount(9)

    .category("category1")

tracker.track(event)
```

**Android (Java):**

```java
MessageNotification event = new MessageNotification("title", "body", MessageNotificationTrigger.push)

        .notificationTimestamp("2021-10-18T10:16:08.008Z")

        .category("category")

        .action("action")

        .bodyLocKey("loc key")

        .bodyLocArgs(Arrays.asList("loc arg1", "loc arg2"))

        .sound("chime.mp3")

        .notificationCount(9)

        .category("category1");

tracker.track(event);
```

***

### Tracking Deep Links

The Deep Link is received by the mobile operating system and passed to the related app. Our mobile tracker can't automatically track the Deep Link, but we provide an out-of-the-box event that can be used by the developer to manually track it as soon as the Deep Link is received in the app.

The event schema is `iglu:com.snowplowanalytics.mobile/deep_link_received/jsonschema/1-0-0`.

It will be the duty of the tracker to automatically attach the information of the Deep Link to the first ScreenView tracked.

In practice, when the app receives a Deep Link, the developer can track it through the DeepLinkReceived event:

**iOS:**

```swift
public func application(_ application: UIApplication,

                        continue userActivity: NSUserActivity,

                        restorationHandler: @escaping ([Any]?) -> Void) -> Bool

{

    ...

    if let url = userActivity.webpageURL {

        let deepLinkEvent = DeepLinkReceived(url: userActivity.webpageURL.absoluteString)

            .referrer(userActivity.referrerURL.absoluteString)

        tracker.track(deepLinkEvent)

    }

    ...

}
```

**Android (Kotlin):**

```kotlin
override fun onCreate(savedInstanceState: Bundle?) {

    ...



    // Extract info from Intent

    val deepLinkUrl = intent.data.toString()

    var referrer: String? = null

    val extras = intent.extras

    if (extras != null) {

        val referrerUri = extras[Intent.EXTRA_REFERRER] as? Uri

        if (referrerUri != null) {

            referrer = referrerUri.toString()

        }

    }

    // Create and track the event

    val event = DeepLinkReceived(deepLinkUrl).referrer(referrer)

    tracker.track(event)



    ...

}
```

**Android (Java):**

```java
@Override

public void onCreate(Bundle savedInstanceState) {

    ...



    // Extract info from Intent

    Intent intent = getIntent();

    String deepLinkUrl = intent.getData().toString();

    String referrer = null;

    Bundle extras = intent.getExtras();

    if (extras != null) {

        Uri referrerUri = extras.get(Intent.EXTRA_REFERRER);

        if (referrerUri != null) {

            referrer = referrerUri.toString();

        }

    }

    // Create and track the event

    DeepLinkReceived event = new DeepLinkReceived(deepLinkUrl).referrer(referrer);

    tracker.track(event);



    ...

}
```

***

The tracker keeps memory of the tracked Deep Link event and will attach a Deep Link entity to the first ScreenView tracked in the tracker. This is helpful during the analysis of the data because it will be clear the relation between the content visualized by the user (ScreenView event) and source (DeepLink entity) that originated that visualisation.

This behavior is enabled by default but it can be disabled from the `TrackerConfiguration`.

For example:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    ...

    .deepLinkContext(false)

    ...
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    ...

    .deepLinkContext(false)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    ...

    .deepLinkContext(false);
```

***

The `DeepLinkReceived` event can be used in pair with a `campaign-attribution-enrichment` appropriately enabled in the Snowplow pipeline. It works exactly like for `PageView` events in the web/JS tracker. When the user taps on an advertising banner or a marketing email or message, it can trigger the launch of the app through the Deep Linking feature. The referral from the advertising campaigns, websites, or other source can be composed by UTM parameters used to attribute the user activity back to the campaign. The Campaign Attribution Enrichment can parse the DeepLinkReceived event extracting the UTM parameters in the deep link url.

The tracked `DeepLinkReceived` event and each subsequent `ScreenView` event also have the URL and referrer information added in the `page_url` and `page_referrer` atomic properties. This makes them compatible with data models and enrichments built for events tracked on the Web.

### Creating a Structured event

Our philosophy in creating Snowplow is that users should design suitable data structures customised for their own consumer interactions data capture. You can read more about that philosophy [here](/docs/event-studio/).

However, as part of a Snowplow implementation there may be interactions where [custom `SelfDescribing` events](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) are unwarranted. Those are candidates to track as `Structured` events. We do strongly recommend using fully custom `SelfDescribing` events.

**iOS:**

```swift
let event = Structured(category: "Example", action: "my-action")

    .label("my-label")

    .property("my-property")

    .value(5)



tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = Structured("shop", "add-to-basket") // category and action

    .label("Add To Basket")

    .property("pcs")

    .value(2.0)

tracker.track(event)
```

**Android (Java):**

```java
Structured event = new Structured("shop", "add-to-basket") // category and action

    .label("Add To Basket")

    .property("pcs")

    .value(2);

tracker.track(event);
```

***

## Tracking data that is not event-type specific

Some data, such as that relating to the user whose activity is being tracked, is relevant across all event types. The tracker provides two mechanisms for tracking this kind of data.

Certain properties, including `userId` or `ipAddress`, can be set as "atomic" properties in the raw event, using the [`Subject` class](/docs/sources/mobile-trackers/client-side-properties/).

A more general and powerful method is to attach self-describing JSON "context entities" to your events - the same JSON schemas as used for self-describing events. This means that any data that can be described by a JSON schema can be added to any or all of your events. Read more about custom tracking [here](/docs/sources/mobile-trackers/custom-tracking-using-schemas/).

All events also provide the option for setting a custom timestamp, called `trueTimestamp`. See below for details.

### Adding custom timestamps to events

Snowplow events have several timestamps. The raw event payload always contains a `deviceCreatedTimestamp` (`dtm`) and a `deviceSentTimestamp` (`stm`). Other timestamps are added as the event moves through the pipeline.

Every `Event` in the tracker allows for a custom timestamp, called `trueTimestamp` to be set. A `trueTimestamp` can be added to any event using the `trueTimestamp()` method:

**iOS:**

```swift
// This example shows a self-describing event, but all events can have a trueTimestamp

let event = SelfDescribing(schema: "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", payload: data)

    .trueTimestamp(Date(timeIntervalSince1970: 166184300))
```

`trueTimestamp` should be a `Date` object.

**Android (Kotlin):**

```kotlin
// This example shows a self-describing event, but all events can have a trueTimestamp

val event = SelfDescribing(SelfDescribingJson("iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", data))

    .trueTimestamp(166184300L)
```

**Android (Java):**

```java
// This example shows a self-describing event, but all events can have a trueTimestamp

SelfDescribing event = new SelfDescribing(new SelfDescribingJson("iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", data))

    .trueTimestamp(166184300L);
```

***

---

# Track application installs with the native mobile trackers
> Track first-time app installations and capture referrer information from Google Play Store for Android apps.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/installation-tracking/

Installation tracking tracks an install event which occurs the first time an application is opened. The tracker will record when it's first been installed, so deleting and reinstalling an app will trigger another install event.

If installation autotracking is not enabled, the tracker will still keep track of when the app was first installed, so that when enabled, the tracker will send the recorded install event with a timestamp reflecting when it was first installed.

The installation autotracking is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .installAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .installAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .installAutotracking(true);
```

***

## Android-only: Tracking referrer information from the Google Play Referrer library

The Google Play Referrer library is a tool provided by Google for Android developers to track and capture referral information when a user installs or updates an app from the Google Play Store. It allows developers to gather valuable insights about the sources that drive app installations, such as ad campaigns, referral links, or marketing efforts.

When an app is installed or updated, the Google Play Referrer library retrieves the referral information from the Google Play Store and provides it to the app. This information includes the referrer URL, which can contain parameters or tracking codes that identify the specific referral source.

The Android tracker can make use of the library to retrieve the referral information. It attaches the information in an entity attached to the install event. The entity uses the `iglu:com.android.installreferrer.api/referrer_details/jsonschema/1-0-0` schema with the following properties:

| Property                 | Type     | Description                                                                                        |
| ------------------------ | -------- | -------------------------------------------------------------------------------------------------- |
| `installReferrer`        | String   | The referrer URL of the installed package                                                          |
| `referrerClickTimestamp` | Datetime | The timestamp when referrer click happens                                                          |
| `installBeginTimestamp`  | Datetime | The timestamp when installation begins                                                             |
| `googlePlayInstantParam` | Boolean  | Boolean indicating if the user has interacted with the app's instant experience in the past 7 days |

To enable tracking the entity, you will need to have `installAutotracking` enabled (it's on by default) and add the following line to the dependencies section of the `build.gradle` file in your app:

```gradle
dependencies {

    ...

    implementation "com.android.installreferrer:installreferrer:2.2"

}
```

---

# Track application lifecycle changes with the native mobile trackers
> Automatically track application foreground and background transitions to understand app usage patterns.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/

The tracker can capture application lifecycle state changes. In particular, when the app changes state from foreground to background and vice versa.

The lifecycle tracking is enabled by default (since v6.0.0). It can be configured in `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .lifecycleAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .lifecycleAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .lifecycleAutotracking(true);
```

***

Once enabled, the tracker will automatically track a [`Background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event) when the app is moved to background and a [`Foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event) when the app moves back to foreground (becomes visible in the screen).

The tracker attaches a [`LifecycleEntity`](/docs/events/ootb-data/mobile-lifecycle-events/#lifecycle-entity) to all the events tracked by the tracker reporting if the app was visible (foreground state) when the event was tracked.

The `LifecycleEntity` value is conditioned by the internal state of the tracker only. To make an example, if the app is in foreground state but the developer tracks a `Background` event intentionally, it would force the generation of a `LifecycleEntity` that mark the app as non visible, even if it's actually visible in the device.

---

# Track media events with the native mobile trackers
> Track media playback events including play, pause, buffering, and completion using the mobile media tracking plugin.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/media-tracking/

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

**iOS:**

```swift
// 1. Start tracking when player loads

let id = UUID().uuidString

let player = MediaPlayerEntity()

  .duration(300)

  .label("My Video Title")

  .mediaType(.video)



let mediaTracking = tracker.media.startMediaTracking(id: id, player: player)



// 2. Update player state every second (or use AVPlayer auto-tracking)

mediaTracking.update(player: MediaPlayerEntity().currentTime(currentTime))



// 3. Track events as they occur

mediaTracking.track(MediaPlayEvent())

mediaTracking.track(MediaPauseEvent())

mediaTracking.track(MediaSeekStartEvent())

mediaTracking.track(MediaSeekEndEvent())

mediaTracking.track(MediaEndEvent())



// 4. End tracking when done

tracker.media.endMediaTracking(id: id)
```

**Android (Kotlin):**

```kotlin
// 1. Start tracking when player loads

val id = UUID.randomUUID().toString()

val player = MediaPlayerEntity(

  duration = 300.0,

  label = "My Video Title",

  mediaType = MediaType.Video

)

val mediaTracking = tracker?.media?.startMediaTracking(id, player)



// 2. Update player state every second

mediaTracking?.update(player = MediaPlayerEntity(currentTime = currentTime))



// 3. Track events as they occur

mediaTracking?.track(MediaPlayEvent())

mediaTracking?.track(MediaPauseEvent())

mediaTracking?.track(MediaSeekStartEvent())

mediaTracking?.track(MediaSeekEndEvent())

mediaTracking?.track(MediaEndEvent())



// 4. End tracking when done

tracker?.media?.endMediaTracking(id)
```

**Android (Java):**

```java
// 1. Start tracking when player loads

String id = UUID.randomUUID().toString();

MediaPlayerEntity player = new MediaPlayerEntity();

player.setDuration(300.0);

player.setLabel("My Video Title");

player.setMediaType(MediaType.Video);



TrackerController tracker = Snowplow.getDefaultTracker();

MediaTracking mediaTracking = tracker.getMedia().startMediaTracking(id, player);



// 2. Update player state every second

MediaPlayerEntity update = new MediaPlayerEntity();

update.setCurrentTime(currentTime);

mediaTracking.update(update, null, null);



// 3. Track events as they occur

mediaTracking.track(new MediaPlayEvent(), null, null, null);

mediaTracking.track(new MediaPauseEvent(), null, null, null);

mediaTracking.track(new MediaSeekStartEvent(), null, null, null);

mediaTracking.track(new MediaSeekEndEvent(), null, null, null);

mediaTracking.track(new MediaEndEvent(), null, null, null);



// 4. End tracking when done

tracker.getMedia().endMediaTracking(id);
```

***

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

**iOS:**

```swift
let player = MediaPlayerEntity()

  .currentTime(0)          // Current playback position in seconds

  .duration(300)           // Total duration in seconds

  .ended(false)            // Whether playback has ended

  .livestream(false)       // Whether this is a live stream

  .label("My Video")       // Human-readable title (recommended)

  .loop(false)             // Whether to restart after ending

  .mediaType(.video)       // .video or .audio

  .muted(false)            // Whether audio is muted

  .paused(true)            // Whether playback is paused

  .pictureInPicture(false) // Whether in picture-in-picture mode

  .playbackRate(1.0)       // Playback speed (1 = normal)

  .quality("1080p")        // Quality level

  .volume(100)             // Volume percentage (0-100)



let mediaTracking = tracker.media.startMediaTracking(id: id, player: player)
```

**Android (Kotlin):**

```kotlin
val player = MediaPlayerEntity(

  currentTime = 0.0,        // Current playback position in seconds

  duration = 300.0,         // Total duration in seconds

  ended = false,            // Whether playback has ended

  livestream = false,       // Whether this is a live stream

  label = "My Video",       // Human-readable title (recommended)

  loop = false,             // Whether to restart after ending

  mediaType = MediaType.Video, // MediaType.Video or MediaType.Audio

  muted = false,            // Whether audio is muted

  paused = true,            // Whether playback is paused

  pictureInPicture = false, // Whether in picture-in-picture mode

  playbackRate = 1.0,       // Playback speed (1 = normal)

  quality = "1080p",        // Quality level

  volume = 100              // Volume percentage (0-100)

)



val mediaTracking = tracker?.media?.startMediaTracking(id, player)
```

**Android (Java):**

```java
MediaPlayerEntity player = new MediaPlayerEntity();

player.setCurrentTime(0.0);          // Current playback position in seconds

player.setDuration(300.0);           // Total duration in seconds

player.setEnded(false);              // Whether playback has ended

player.setLivestream(false);         // Whether this is a live stream

player.setLabel("My Video");         // Human-readable title (recommended)

player.setLoop(false);               // Whether to restart after ending

player.setMediaType(MediaType.Video); // MediaType.Video or MediaType.Audio

player.setMuted(false);              // Whether audio is muted

player.setPaused(true);              // Whether playback is paused

player.setPictureInPicture(false);   // Whether in picture-in-picture mode

player.setPlaybackRate(1.0);         // Playback speed (1 = normal)

player.setQuality("1080p");          // Quality level

player.setVolume(100);               // Volume percentage (0-100)



MediaTracking mediaTracking = tracker.getMedia().startMediaTracking(id, player);
```

***

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id)

  .pingInterval(30)   // Seconds between pings (default: 30)

  .maxPausedPings(1)  // Max pings while paused (default: 1)



// Or turn off pings entirely

let configuration = MediaTrackingConfiguration(id: id)

  .pings(false)
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(

  id = id,

  pingInterval = 30,   // Seconds between pings (default: 30)

  maxPausedPings = 1   // Max pings while paused (default: 1)

)



// Or turn off pings entirely

val configuration = MediaTrackingConfiguration(id = id, pings = false)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);

configuration.setPingInterval(30);   // Seconds between pings (default: 30)

configuration.setMaxPausedPings(1);  // Max pings while paused (default: 1)



// Or turn off pings entirely

configuration.setPings(false);
```

***

### Percentage progress

Track events when playback reaches specified percentage boundaries.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id)

  .boundaries([10, 25, 50, 75, 90])  // Fire events at these percentages
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(

  id = id,

  boundaries = listOf(10, 25, 50, 75, 90)  // Fire events at these percentages

)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);

configuration.setBoundaries(Arrays.asList(10, 25, 50, 75, 90));  // Fire events at these percentages
```

***

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id).session(true)
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(id = id, session = true)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);

configuration.setSession(true);
```

***

### Custom entities

You can attach custom entities to all events for this media tracking instance.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id)

  .entities([

      SelfDescribingJson(

          schema: "iglu:com.example/video_metadata/jsonschema/1-0-0",

          andData: ["contentId": "abc123", "category": "tutorial"]

      )

  ])
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(

  id = id,

  entities = listOf(

      SelfDescribingJson(

          schema = "iglu:com.example/video_metadata/jsonschema/1-0-0",

          data = mapOf("contentId" to "abc123", "category" to "tutorial")

      )

  )

)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);

configuration.setEntities(Collections.singletonList(

  new SelfDescribingJson(

      "iglu:com.example/video_metadata/jsonschema/1-0-0",

      new HashMap<String, Object>() {{

          put("contentId", "abc123");

          put("category", "tutorial");

      }}

  )

));
```

***

### Filter events

**iOS:**

Only track specific event types by providing an allowlist:

```swift
let configuration = MediaTrackingConfiguration(id: id)

  .captureEvents([MediaPlayEvent.self, MediaPauseEvent.self, MediaEndEvent.self])
```

**Android (Kotlin):**

Only track specific event types by providing an allowlist:

```kotlin
val configuration = MediaTrackingConfiguration(

  id = id,

  captureEvents = listOf(MediaPlayEvent::class, MediaPauseEvent::class, MediaEndEvent::class)

)
```

**Android (Java):**

Only track specific event types by providing an allowlist:

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);

configuration.setCaptureEvents(Arrays.asList(

  MediaPlayEvent.class, MediaPauseEvent.class, MediaEndEvent.class

));
```

***

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

**iOS:**

```swift
mediaTracking.track(MediaReadyEvent())        // Player is ready

mediaTracking.track(MediaPlayEvent())         // Playback started

mediaTracking.track(MediaPauseEvent())        // Playback paused

mediaTracking.track(MediaEndEvent())          // Playback ended

mediaTracking.track(MediaSeekStartEvent())    // Seek operation started

mediaTracking.track(MediaSeekEndEvent())      // Seek operation ended

mediaTracking.track(MediaBufferStartEvent())  // Buffering started

mediaTracking.track(MediaBufferEndEvent())    // Buffering ended
```

**Android (Kotlin):**

```kotlin
mediaTracking?.track(MediaReadyEvent())        // Player is ready

mediaTracking?.track(MediaPlayEvent())         // Playback started

mediaTracking?.track(MediaPauseEvent())        // Playback paused

mediaTracking?.track(MediaEndEvent())          // Playback ended

mediaTracking?.track(MediaSeekStartEvent())    // Seek operation started

mediaTracking?.track(MediaSeekEndEvent())      // Seek operation ended

mediaTracking?.track(MediaBufferStartEvent())  // Buffering started

mediaTracking?.track(MediaBufferEndEvent())    // Buffering ended
```

**Android (Java):**

```java
mediaTracking.track(new MediaReadyEvent(), null, null, null);        // Player is ready

mediaTracking.track(new MediaPlayEvent(), null, null, null);         // Playback started

mediaTracking.track(new MediaPauseEvent(), null, null, null);        // Playback paused

mediaTracking.track(new MediaEndEvent(), null, null, null);          // Playback ended

mediaTracking.track(new MediaSeekStartEvent(), null, null, null);    // Seek operation started

mediaTracking.track(new MediaSeekEndEvent(), null, null, null);      // Seek operation ended

mediaTracking.track(new MediaBufferStartEvent(), null, null, null);  // Buffering started

mediaTracking.track(new MediaBufferEndEvent(), null, null, null);    // Buffering ended
```

***

### Player state events

**iOS:**

```swift
mediaTracking.track(MediaPlaybackRateChangeEvent(newRate: 1.5))

mediaTracking.track(MediaVolumeChangeEvent(newVolume: 80))

mediaTracking.track(MediaFullscreenChangeEvent(fullscreen: true))

mediaTracking.track(MediaPictureInPictureChangeEvent(pictureInPicture: true))

mediaTracking.track(MediaQualityChangeEvent(

  newQuality: "1080p",

  bitrate: 5000,

  framesPerSecond: 30,

  automatic: true

))

mediaTracking.track(MediaErrorEvent(

  errorCode: "500",

  errorName: "NetworkError",

  errorDescription: "Failed to load media"

))
```

**Android (Kotlin):**

```kotlin
mediaTracking?.track(MediaPlaybackRateChangeEvent(newRate = 1.5))

mediaTracking?.track(MediaVolumeChangeEvent(newVolume = 80))

mediaTracking?.track(MediaFullscreenChangeEvent(fullscreen = true))

mediaTracking?.track(MediaPictureInPictureChangeEvent(pictureInPicture = true))

mediaTracking?.track(MediaQualityChangeEvent(

  newQuality = "1080p",

  bitrate = 5000,

  framesPerSecond = 30,

  automatic = true

))

mediaTracking?.track(MediaErrorEvent(

  errorCode = "500",

  errorName = "NetworkError",

  errorDescription = "Failed to load media"

))
```

**Android (Java):**

```java
mediaTracking.track(new MediaPlaybackRateChangeEvent(null, 1.5), null, null, null);

mediaTracking.track(new MediaVolumeChangeEvent(null, 80), null, null, null);

mediaTracking.track(new MediaFullscreenChangeEvent(true), null, null, null);

mediaTracking.track(new MediaPictureInPictureChangeEvent(true), null, null, null);



MediaQualityChangeEvent qualityEvent = new MediaQualityChangeEvent();

qualityEvent.setNewQuality("1080p");

qualityEvent.setBitrate(5000);

qualityEvent.setFramesPerSecond(30);

qualityEvent.setAutomatic(true);

mediaTracking.track(qualityEvent, null, null, null);



MediaErrorEvent errorEvent = new MediaErrorEvent();

errorEvent.setErrorCode("500");

errorEvent.setErrorName("NetworkError");

errorEvent.setErrorDescription("Failed to load media");

mediaTracking.track(errorEvent, null, null, null);
```

***

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

**iOS:**

```swift
// Ad break events

let adBreak = MediaAdBreakEntity(breakId: "break-1")

  .name("pre-roll")

  .breakType(.linear)

  .podSize(3)

mediaTracking.track(MediaAdBreakStartEvent(), adBreak: adBreak)

mediaTracking.track(MediaAdBreakEndEvent())



// Ad events

let ad = MediaAdEntity(adId: "ad-1")

  .name("Product Ad")

  .creativeId("creative-1")

  .duration(30)

  .skippable(true)

mediaTracking.track(MediaAdStartEvent(), ad: ad)

mediaTracking.track(MediaAdFirstQuartileEvent())

mediaTracking.track(MediaAdMidpointEvent())

mediaTracking.track(MediaAdThirdQuartileEvent())

mediaTracking.track(MediaAdCompleteEvent())

mediaTracking.track(MediaAdSkipEvent())

mediaTracking.track(MediaAdClickEvent())

mediaTracking.track(MediaAdPauseEvent())

mediaTracking.track(MediaAdResumeEvent())
```

**Android (Kotlin):**

```kotlin
// Ad break events

val adBreak = MediaAdBreakEntity(

  breakId = "break-1",

  name = "pre-roll",

  breakType = MediaAdBreakType.Linear,

  podSize = 3

)

mediaTracking?.track(MediaAdBreakStartEvent(), adBreak = adBreak)

mediaTracking?.track(MediaAdBreakEndEvent())



// Ad events

val ad = MediaAdEntity(

  adId = "ad-1",

  name = "Product Ad",

  creativeId = "creative-1",

  duration = 30.0,

  skippable = true

)

mediaTracking?.track(MediaAdStartEvent(), ad = ad)

mediaTracking?.track(MediaAdFirstQuartileEvent())

mediaTracking?.track(MediaAdMidpointEvent())

mediaTracking?.track(MediaAdThirdQuartileEvent())

mediaTracking?.track(MediaAdCompleteEvent())

mediaTracking?.track(MediaAdSkipEvent())

mediaTracking?.track(MediaAdClickEvent())

mediaTracking?.track(MediaAdPauseEvent())

mediaTracking?.track(MediaAdResumeEvent())
```

**Android (Java):**

```java
// Ad break events

MediaAdBreakEntity adBreak = new MediaAdBreakEntity("break-1");

adBreak.setName("pre-roll");

adBreak.setBreakType(MediaAdBreakType.Linear);

adBreak.setPodSize(3);

mediaTracking.track(new MediaAdBreakStartEvent(), null, null, adBreak);

mediaTracking.track(new MediaAdBreakEndEvent(), null, null, null);



// Ad events

MediaAdEntity ad = new MediaAdEntity("ad-1");

ad.setName("Product Ad");

ad.setCreativeId("creative-1");

ad.setDuration(30.0);

ad.setSkippable(true);

mediaTracking.track(new MediaAdStartEvent(), null, ad, null);

mediaTracking.track(new MediaAdFirstQuartileEvent(), null, null, null);

mediaTracking.track(new MediaAdMidpointEvent(), null, null, null);

mediaTracking.track(new MediaAdThirdQuartileEvent(), null, null, null);

mediaTracking.track(new MediaAdCompleteEvent(), null, null, null);

mediaTracking.track(new MediaAdSkipEvent(), null, null, null);

mediaTracking.track(new MediaAdClickEvent(), null, null, null);

mediaTracking.track(new MediaAdPauseEvent(), null, null, null);

mediaTracking.track(new MediaAdResumeEvent(), null, null, null);
```

***

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

**iOS:**

```swift
mediaTracking.track(SelfDescribing(

  schema: "iglu:com.example/video_interaction/jsonschema/1-0-0",

  payload: ["action": "share", "platform": "twitter"]

))
```

**Android (Kotlin):**

```kotlin
mediaTracking?.track(SelfDescribing(

  "iglu:com.example/video_interaction/jsonschema/1-0-0",

  mapOf("action" to "share", "platform" to "twitter")

))
```

**Android (Java):**

```java
mediaTracking.track(new SelfDescribing(

  "iglu:com.example/video_interaction/jsonschema/1-0-0",

  new HashMap<String, Object>() {{

      put("action", "share");

      put("platform", "twitter");

  }}

), null, null, null);
```

***

## Update player state

**iOS:**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```swift
mediaTracking.update(player: MediaPlayerEntity().currentTime(currentTime))
```

**Android (Kotlin):**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```kotlin
mediaTracking?.update(player = MediaPlayerEntity(currentTime = currentTime))
```

**Android (Java):**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```java
MediaPlayerEntity player = new MediaPlayerEntity();

player.setCurrentTime(currentTime);

mediaTracking.update(player, null, null);
```

***

## AVPlayer auto-tracking (iOS)

The iOS tracker can automatically track events from [AVPlayer](https://developer.apple.com/documentation/avfoundation/avplayer). Pass the player instance to `startMediaTracking`:

```swift
let mediaTracking = tracker.media.startMediaTracking(

    player: avPlayer,

    configuration: MediaTrackingConfiguration(id: "my-video")

)
```

The following events are auto-tracked:

- Play events
- Pause events
- Seek end events
- Ping events
- Percent progress events (if configured)
- Buffer start events
- End events
- Error events

---

# Track platform and application data with the native mobile trackers
> Automatically capture device and application context including OS version, device model, battery level, and advertising identifiers.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/

Platform and application data tracking features capture information about the device and the app.

They are enabled by default. But the setting can be changed through `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .platformContext(true)

    .applicationContext(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .platformContext(true)

    .applicationContext(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .platformContext(true)

    .applicationContext(true);
```

***

## Application context

The [application context entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.mobile/application/jsonschema/1-0-0) contains two properties:

| Property  | Type   | Description                                             | Required in schema |
| --------- | ------ | ------------------------------------------------------- | ------------------ |
| `version` | String | Version number of the application e.g 1.1.0             | Yes                |
| `build`   | String | Build name of the application e.g s9f2k2d or 1.1.0 beta | Yes                |

## Platform (mobile) context

The [platform context entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-3) contains the following properties:

| Property                | Type                                 | Android | iOS | Description                                                                                                                                                             | Required in schema |
| ----------------------- | ------------------------------------ | ------- | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `osType`                | String                               | ✅       | ✅   | Type of the operating system (e.g., "ios", "tvos", "watchos", "osx", "android")                                                                                         | Yes                |
| `osVersion`             | String                               | ✅       | ✅   | Version of the mobile operating system                                                                                                                                  | Yes                |
| `deviceManufacturer`    | String                               | ✅       | ✅   | Device vendor                                                                                                                                                           | Yes                |
| `deviceModel`           | String                               | ✅       | ✅   | Model of the device                                                                                                                                                     | Yes                |
| `carrier`               | String                               | ✅       | ✅   | Carrier of the SIM inserted in the device                                                                                                                               | No                 |
| `networkType`           | String                               | ✅       | ✅   | One of: "mobile", "wifi", "offline"                                                                                                                                     | No                 |
| `networkTechnology`     | String                               | ✅       | ✅   | Radio access technology that the device is using                                                                                                                        | No                 |
| `openIdfa`              | String                               | ❌       | ❌   | Deprecated property                                                                                                                                                     | No                 |
| `appleIdfa`             | String                               | ❌       | ✅   | Advertising identifier on iOS                                                                                                                                           | No                 |
| `appleIdfv`             | String                               | ❌       | ✅   | UUID [identifier for vendors](https://developer.apple.com/documentation/uikit/uidevice/1620059-identifierforvendor) on iOS                                              | No                 |
| `androidIdfa`           | String                               | ✅       | ❌   | Advertising identifier on Android                                                                                                                                       | No                 |
| `physicalMemory`        | Integer                              | ✅       | ✅   | Total physical system memory in bytes                                                                                                                                   | No                 |
| `systemAvailableMemory` | Integer                              | ✅       | ❌   | Available memory on the system in bytes                                                                                                                                 | No                 |
| `appAvailableMemory`    | Integer                              | ❌       | ❌   | Amount of memory in bytes available to the current app                                                                                                                  | No                 |
| `batteryLevel`          | Integer                              | ✅       | ✅   | Remaining battery level as an integer percentage of total battery capacity                                                                                              | No                 |
| `batteryState`          | String                               | ✅       | ✅   | Battery state for the device. One of: "unplugged", "charging", "full"                                                                                                   | No                 |
| `lowPowerMode`          | Boolean                              | ❌       | ✅   | A Boolean indicating whether Low Power Mode is enabled                                                                                                                  | No                 |
| `availableStorage`      | Integer                              | ✅       | ❌   | Bytes of storage remaining                                                                                                                                              | No                 |
| `totalStorage`          | Integer                              | ✅       | ❌   | Total size of storage in bytes                                                                                                                                          | No                 |
| `isPortrait`            | Boolean                              | ✅       | ✅   | A Boolean indicating whether the device orientation is portrait (either upright or upside down)                                                                         | No                 |
| `resolution`            | String                               | ✅       | ✅   | Screen resolution in pixels. Arrives in the form of WIDTHxHEIGHT (e.g., 1200x900). Doesn't change when device orientation changes. See note below.                      | No                 |
| `scale`                 | Number                               | ✅       | ✅   | Scale factor used to convert logical coordinates to device coordinates of the screen (uses UIScreen.scale on iOS and DisplayMetrics.density on Android)                 | No                 |
| `language`              | String                               | ✅       | ✅   | System language currently used on the device (ISO 639)                                                                                                                  | No                 |
| `appSetId`              | String                               | ✅       | ❌   | Android vendor ID scoped to the set of apps published under the same Google Play developer account (see <https://developer.android.com/training/articles/app-set-id>)   | No                 |
| `appSetIdScope`         | String (either "app" or "developer") | ✅       | ❌   | Scope of the `appSetId`. Can be scoped to the app or to a developer account on an app store (all apps from the same developer on the same device will have the same ID) | No                 |

> **Note:** The screen resolution for the platform entity is obtained from the Android context resources. The height value will likely be lower than that reported in the canonical `dvce_screenheight` [event property](/docs/fundamentals/canonical-event/#device-fields), which is fetched from `WindowManager`, an older API that still includes the menu bar.
>
> To standardize the screen resolution between event and entity properties, provide a `SubjectConfiguration` with `useContextResourcesScreenResolution(true)` flag at tracker initialization. This flag is false by default, and available from Android tracker v6.0.3 onwards. Read about configuring `Subject` properties [here](/docs/sources/mobile-trackers/client-side-properties/).

### Choosing which properties to track

You can choose which properties should be added to the platform context entity. By default, all available properties are tracked in the entity. In case you don't want certain properties to be tracked, you can choose the ones to track using `TrackerConfiguration.platformContextEntities`:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .platformContextProperties([.batteryLevel, .isPortrait, .language])
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

     .platformContextProperties(listOf(PlatformContextProperty.BATTERY_LEVEL, PlatformContextProperty.IS_PORTRAIT))
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .platformContextProperties(Collections.singletonList(PlatformContextProperty.BATTERY_LEVEL, PlatformContextProperty.IS_PORTRAIT));
```

***

### Overriding platform context properties

In case you want to override the values for certain properties of the platform context (or provide ones that are not tracked by default, such as the `totalStorage` on iOS), you can set the `platformContextRetriever` in `TrackerConfiguration` to provide these values.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .platformContextRetriever(

        PlatformContextRetriever(deviceVendor: { "my-custom-vendor" })

    )
```

**Android (Kotlin):**

```kotlin
val trackerConfiguration = TrackerConfiguration("appId")

    .platformContextRetriever(

        PlatformContextRetriever(deviceVendor = { "my-custom-vendor" })

    )
```

**Android (Java):**

```java
PlatformContextRetriever platformContextRetriever = new PlatformContextRetriever();

platformContextRetriever.setDeviceVendor(() -> "my-custom-vendor");

TrackerConfiguration trackerConfiguration = new TrackerConfiguration("appId")

    .platformContextRetriever(platformContextRetriever);
```

***

### Identifier for Advertisers (IDFA/AAID)

The IDFA advertising identifiers are only added to the platform context if you fulfill the following requirements. Otherwise, their values will be NULL.

**iOS:**

The Apple advertising identifier is stored in the `appleIdfa` property.

Starting with iOS 14, one has to request user consent through the [App Tracking Transparency](https://developer.apple.com/documentation/apptrackingtransparency) framework in order to access the advertising identifier.

1. Add and follow the guidelines of [App Tracking Transparency Framework](https://developer.apple.com/documentation/apptrackingtransparency) in your app.
2. Pass a callback to your `TrackerConfiguration` that retrieves the identifier:

```swift
import AdSupport



let tracker = Snowplow.createTracker(namespace: "ns", network: networkConfig) {

    TrackerConfiguration()

        .advertisingIdentifierRetriever {

            ASIdentifierManager.shared().advertisingIdentifier

        }

}
```

> **Note:** The simulators can’t generate a proper IDFA, instead they generate a sequence of zeros. If you want to test IDFA with a real code, please use the physical device.
>
> The user has the ability to limit ad-tracking from the device’s Settings. If the user enable the limitations the tracker will not be able to track the IDFA.

**Android:**

The AAID (Android Advertising ID) is a unique user-resettable identifier, which uniquely identifies a particular user for advertising use cases, such as ad personalization. The tracker allows retrieval of the AAID, sending it as property `androidIdfa`.

For privacy purposes the user can reset the identifier at any moment. In that case the tracker will report a new AAID, despite the device and user being the same as before. Also, the user can "Opt out of Ads Personalisation" from the Android settings menu. In that case the tracker will report an empty string in place of the AAID.

If you want to track the AAID, you need to add the Google Mobile Ads library to your app. If it isn’t included, the tracker will not send the AAID with the events.

The Google Mobile Ads can be imported in the `dependencies` section of the `build.gradle` adding:

```gradle
dependencies {

    ...

    implementation 'com.google.android.gms:play-services-ads:22.6.0'

    ...

}
```

The Google Mobile Ads SDK v.17.0.0 introduced some [changes](https://ads-developers.googleblog.com/2018/10/announcing-v1700-of-android-google.html) requiring a tag in the `androidManifest.xml` explained below.

#### Manifest tag for AdMob publishers

AdMob publishers have to add the AdMob app ID in the `AndroidManifest.xml` file:

```xml
<manifest>

    <application>

        <!-- TODO: Replace with your real AdMob app ID -->

        <meta-data

            android:name="com.google.android.gms.ads.APPLICATION_ID"

            android:value="ca-app-pub-################~##########"/>

    </application>

</manifest>
```

Failure to add this tag will result in the app crashing at app launch with a message starting with "The Google Mobile Ads SDK was initialized incorrectly".

#### Manifest tag for Google Ad Manager publishers

Publishers using Google Ad Manager have to declare the app an "Ad Manager app" in the `AndroidManifest.xml` file:

```xml
<manifest>

    <application>

        <meta-data

            android:name="com.google.android.gms.ads.AD_MANAGER_APP"

            android:value="true"/>

    </application>

</manifest>
```

Failure to add this tag will result in the app crashing at app launch with a message starting with "The Google Mobile Ads SDK was initialized incorrectly".

Read more about [Google Play Data safety here.](/docs/sources/mobile-trackers/android-google-play-data-safety/)

***

### App set ID (Android only)

To identify a set of apps owned by an organization, Google provides the app set ID. Read more about it [here](https://developer.android.com/training/articles/app-set-id).

An extra dependency is required to populate the Android-specific properties `appSetId` and `appSetIdScope` within the platform context entity.

```gradle
dependencies {

    ...

    implementation 'com.google.android.gms:play-services-appset:16.0.2'

    ...

}
```

---

# Track screen views and engagement with the native mobile trackers
> Automatically track screen changes, time spent on screens, and user engagement metrics in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/screen-tracking/

Screen view tracking captures screen changes within the app.

**iOS:**

## Screen View Tracking in UIKit

The screen view tracking for UIKit views is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

```swift
let trackerConfig = TrackerConfiguration()

    .screenViewAutotracking(true)

    .screenContext(true)
```

Using method swizzling in the `ViewController` class, the tracker automatically detects when screens are loaded (triggered by viewDidAppear in a ViewController) and tracks events that include information about the current and previous view controllers.

## Screen View tracking in SwiftUI

The tracker is able to track screen view events when selected view components appear in the SwiftUI lifecycle. To provide this functionality, it implements an extension over the `View` component which lets you annotate the components the events should be tracked for using the `snowplowScreen()` function:

```swift
import SwiftUI



struct ProductList: View {

    var body: some View {

        List {

            ...

        }

        .snowplowScreen(name: "ProductList") // this will ensure that screen view events are tracked for this view

    }

}
```

You can provide additional context entities to be tracked with the screen view events and also choose the tracker namespace to use:

```swift
import SwiftUI



struct ProductDetail: View {

    var product: Product



    var body: some View {

        List {

            ...

        }

        .snowplowScreen(

            name: "ProductDetail",

            entities: [ // list of context entities attached to the events

                (

                    schema: "iglu:com.acme_company/example/jsonschema/2-1-1",

                    data: [ "name": product.name ]

                )

            ],

            trackerNamespace: "ns" // namespace of tracker to track the event with

        )

    }

}
```

**Android (Kotlin):**

## Screen View tracking in traditional Activity-based apps

The screen view tracking for Activities (screens) is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .screenViewAutotracking(true)

    .screenContext(true)
```

Using the Android `Application.ActivityLifecycleCallbacks` interface, the tracker automatically detects when Activities are loaded and tracks events that include information about the current and previous Activity.

## Screen View tracking in Jetpack Compose apps

Apps built with Jetpack Compose are constructed from Composable functions rather than Activities, and so the built-in ScreenViewAutotracking will not work.

One possibility for an equivalent functionality is to add a navigation listener, which can track a screen view for each new destination.

```kotlin
fun autoTrackScreenView(navController: NavController) {

    navController.addOnDestinationChangedListener { _, destination, _ ->

        Snowplow.defaultTracker?.track(ScreenView(destination.route ?: "null"))

    }

}
```

Try out this example in the Compose demo inside the [Android codebase](https://github.com/snowplow/snowplow-android-tracker).

Depending how your app is configured, this listener may result in a `ScreenView` being tracked when the app returns from background. This is not how the tracker is intended to work; our screen engagement feature tracks the time spent on a screen whether in foreground or background. If you are experiencing this extra `ScreenView` (and `ScreenEnd`) being tracked, consider adding a check to the listener callback.

**Android (Java):**

## Screen View tracking in traditional Activity-based apps

The screen view tracking for Activities (screens) is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .screenViewAutotracking(true)

    .screenContext(true);
```

Using the Android `Application.ActivityLifecycleCallbacks` interface, the tracker automatically detects when Activities are loaded and tracks events that include information about the current and previous Activity.

***

## Screen view event and screen context entity

Automatic screen view tracking tracks two pieces of information:

- The tracker automatically tracks each screen change using a [`ScreenView` event](/docs/events/ootb-data/page-and-screen-view-events/#screen-views).
- If the `TrackerConfiguration.screenContext` property is enabled, the tracker attaches a [`Screen` entity](/docs/events/ootb-data/page-and-screen-view-events/#screen-entity) to all the events tracked by the tracker reporting the last (and probably current) screen visible on device when the event was tracked.

The `Screen` entity is conditioned by the internal state of the tracker only. To make an example, if the developer manually tracks a `ScreenView` event, all the following events will have a `Screen` entity attached reporting the same information as the last tracked ScreenView event, even if it was manually tracked and the app is in a different screen.

Indeed, disabling the `screenViewAutotracking` only, the tracker can still attach `Screen` entities automatically based only to the manual tracking of `ScreenView` events, and vice versa.

## Screen engagement tracking

> **Note:** This feature has been added in the version 6.0.0 of the iOS and Android trackers.

Screen engagement tracking is a feature that enables tracking the user activity on the screen. This consists of the time spent and the amount of content viewed on the screen.

Concretely, it consists of the following metrics:

1. Time spent on screen while the app was in foreground (tracked automatically).
2. Time spent on screen while the app was in background (tracked automatically).
3. Number of list items scrolled out of all list items (requires some manual tracking).
4. Scroll depth in pixels (requires some manual tracking).

This information is attached using a [`screen_summary` context entity](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) to the following events:

1. [`screen_end` event](/docs/events/ootb-data/page-activity-tracking/#screen-end-event) that is automatically tracked before a new screen view event.
2. [`application_background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event).
3. [`application_foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event).

Screen engagement tracking is enabled by default, but can be configured using the `TrackerConfiguration.screenEngagementAutotracking` option.

For a demo of how mobile screen engagement tracking works in action, **[please visit this demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/)**.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .screenEngagementAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .screenEngagementAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .screenEngagementAutotracking(true);
```

***

### Screen time

Screen time is tracked automatically by the tracker based on the screen view events.

Additionally, the tracker makes use of `application_background` and `application_foreground` events to distinguish between screen time in foreground (`foreground_sec` property in `screen_summary` entity) and screen time in background (`background_sec` property). Make sure that lifecycle autotracking is enabled (it is by default) in order for the tracker to be able to distinguish between these two states.

The foreground time is translated into the engaged time during modeling with the unified dbt package (see below). Foreground and background time together result in the absolute time on screen.

### List item view tracking

Part of screen engagement is tracking how much users saw on the screen. There are two ways that this information can be tracked:

1. By the number of items in a list viewed on the screen.
2. By the scroll depth on the screen in pixels.

The first approach assumes that there is a single list displayed on the screen, ideally with items of the same size. In this scenario, the track can track the index of the last viewed item and the total count of all items.

To provide the tracker with the information that a new list item was viewed by a user, the app will track a `ListItemView` event. Although it is tracked as an event, the tracker won't send it individually to the collector (as long as `screenEngagementAutotracking` is enabled) but will process the information into the next `screen_summary` entity and discard the list item view event.

**iOS:**

```swift
let event = ListItemView(

    index: 1, // Index of the item in the list.

    totalItems: 10 // Total number of items in the list.

)

Snowplow.defaultTracker()?.track(event)
```

#### Using the `snowplowListItem` modifier in SwiftUI

If you are using SwiftUI in your app, you can use the `snowplowListItem` view modifier to automatically track list item views. Annotating the items in your list will ensure that they are tracked as they become visible.

```swift
import SwiftUI



struct ProductList: View {

    var products: [Product]



    var body: some View {

        List {

            ForEach(Array(products.enumerated()), id: \.1.url) { offset, product in

                NavigationLink {

                    ProductDetail(url: product)

                } label: {

                    ...

                }

                .snowplowListItem(index: offset, itemsCount: products.count) // will track the ListItemView event automatically

            }

        }

        .snowplowScreen(...)

    }

}
```

**Android (Kotlin):**

```kt
val event = ListItemView(

    index = 10, // Index of the item in the list.

    itemsCount = 100 // Total number of items in the list.

)

Snowplow.defaultTracker?.track(event)
```

**Android (Java):**

```java
ListItemView event = new ListItemView(

    10, // Index of the item in the list.

    100 // Total number of items in the list.

);

Snowplow.defaultTracker.track(event);
```

***

### Scroll depth tracking

The second approach for tracking how much users saw on the screen is to track the scroll depth in pixels. This may be useful in case you have a single scroll view on the screen which can be scrolled horizontally, vertically or in both directions.

To give the tracker the information of the scroll depth, one can use the `ScrollChanged` event. Similarly like in the `ListItemView` event, this event won't be send to the collector as an individual event (as long as `screenEngagementAutotracking` is enabled) but will be processed into the `screen_summary` entity tracked with the next lifecycle or screen end event.

The `ScrollChanged` event contains information about the scroll offset as well as the content size (size of the content shown in the scroll view) and the dimensions of the scroll view on the screen. To track the event in UIKit on iOS, for instance, you can make use of the `scrollViewDidEndDecelerating` and `scrollViewDidEndDragging` callbacks in `UIScrollViewDelegate`.

**iOS:**

```swift
let event = ScrollChanged(xOffset: 15, yOffset: 30, viewWidth: 15, viewHeight: 20, contentWidth: 200, contentHeight: 100)

Snowplow.defaultTracker()?.track(event)
```

**Android (Kotlin):**

```kt
val event = ScrollChanged(

    xOffset = 15, // Horizontal scroll offset in pixels.

    yOffset = 30, // Vertical scroll offset in pixels.

    viewWidth = 15, // Width of the scroll view in pixels.

    viewHeight = 20, // Height of the scroll view in pixels.

    contentWidth = 150, // Height of the content of the scroll view in pixels

    contentHeight = 100 // Width of the content of the scroll view in pixels

)

Snowplow.defaultTracker?.track(event)
```

**Android (Java):**

```java
ScrollChanged event = new ScrollChanged();

event.setXOffset(15); // Horizontal scroll offset in pixels.

event.setYOffset(10); // Vertical scroll offset in pixels.

event.setViewHeight(20); // Height of the scroll view in pixels.

event.setViewWidth(30); // Width of the scroll view in pixels.

event.setContentHeight(100); // Height of the content of the scroll view in pixels

event.setContentWidth(150); // Width of the content of the scroll view in pixels

Snowplow.defaultTracker.track(event);
```

***

### Modeled data using the Snowplow Unified dbt package

Starting with version 0.2.0, the [Snowplow Unified Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) will process the screen engagement information in the `screen_summary` entity to provide the following metrics:

1. In the `snowplow_unified_views` derived table:

   - `engaged_time_in_s` will contain the screen foreground time.
   - `absolute_time_in_s` will contain the screen foreground + background time.
   - `last_list_item_index`, `list_items_count`, and `list_items_percentage_scrolled` will contain information about the number of viewed items in a list.
   - `horizontal_pixels_scrolled`, `vertical_pixels_scrolled`, `horizontal_percentage_scrolled`, `horizontal_percentage_scrolled` will contain information about the scroll depth in pixels.

2. In the `snowplow_unified_sessions` derived table:

   - `engaged_time_in_s` will contain the screen foreground time.
   - `absolute_time_in_s` will contain the screen foreground + background time.

3. In the `snowplow_unified_users` derived table:

   - `engaged_time_in_s` will contain the screen foreground time.
   - `absolute_time_in_s` will contain the screen foreground + background time.

---

# Track sessions with the native mobile trackers
> Track user sessions with configurable timeouts, session callbacks, and cross-navigation support in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/session-tracking/

Session tracking captures the session which helps to keep track of the user activity in the app.

Client session tracking is enabled by default. It can be set through the `TrackerConfiguration` as explained below.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()

    .sessionContext(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")

    .sessionContext(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")

    .sessionContext(true);
```

***

When enabled, the tracker appends a [`client_session` entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) to each event it sends and it maintains this session information as long as the application is installed on the device.

Sessions correspond to tracked user activity. User information is set via the [session entity](/docs/events/ootb-data/user-and-session-identification/#session-entity).

A session expires when no tracking events have occurred for the amount of time defined in a timeout (by default 30 minutes). The session timeout check is executed for each event tracked. If the gap between two consecutive events is longer than the timeout the session is renewed. There are two timeouts since a session can timeout in the foreground (while the app is visible) or in the background (when the app has been suspended, but not closed).

The timeouts for the session can be configured in the `SessionConfiguration` like in the example below:

**iOS:**

```swift
let sessionConfig = SessionConfiguration(

    foregroundTimeout: Measurement(value: 360, unit: .seconds),

    backgroundTimeout: Measurement(value: 360, unit: .seconds)

)

Snowplow.createTracker(

    namespace: "appTracker",

    network: networkConfig,

    configurations: [trackerConfig, sessionConfig]

)
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration(

    TimeMeasure(6, TimeUnit.SECONDS),

    TimeMeasure(30, TimeUnit.SECONDS)

)

Snowplow.createTracker(applicationContext, namespace, networkConfig, sessionConfig)
```

**Android (Java):**

```java
SessionConfiguration sessionConfig = new SessionConfiguration(

    new TimeMeasure(6, TimeUnit.SECONDS),

    new TimeMeasure(30, TimeUnit.SECONDS)

);

Snowplow.createTracker(getApplicationContext(), namespace, networkConfig, sessionConfig);
```

***

The lifecycle events (`Foreground` and `Background` events) have a role in the session expiration. The lifecycle events can be enabled as explained in [App Lifecycle Tracking](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/). Once enabled they will be fired automatically when the app moves from foreground state to background state and vice versa.

When the app moves from foreground to background, the `Background` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the foreground timeout. When the app moves from background to foreground, the `Foreground` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the background timeout.

For instance, with this configuration:

**iOS:**

```swift
SessionConfiguration(

    foregroundTimeout: Measurement(value: 360, unit: .seconds),

    backgroundTimeout: Measurement(value: 15, unit: .seconds)

)
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration(

    TimeMeasure(360, TimeUnit.SECONDS),

    TimeMeasure(15, TimeUnit.SECONDS)

)
```

**Android (Java):**

```java
SessionConfiguration sessionConfig = new SessionConfiguration(

    new TimeMeasure(360, TimeUnit.SECONDS),

    new TimeMeasure(15, TimeUnit.SECONDS)

);
```

***

the session would expire if the app is in background for more than 15 seconds, like in this example:

```text
time: 0s - ScreenView event - foreground timeout session check - session 1

time: 3s - Background event - foreground timeout session check (3 < 360) - session 1

time: 30s - Foreground event - background timeout session check (30 > 15) - session 2
```

In the above example, the `Foreground` event triggers a new session because the time spent in background (without tracked events) is bigger than the background timeout for the session.

## Session callback

> **Info:** This feature is available since v3.1.

The tracker allows the configuration of a callback to inform the app every time a new session is created (in correspondence of a session timeout check). This can be configured in the `SessionConfiguration` and it provides the `SessionState` where all the info already tracked in the session can be accessed.

Below is an example of where the session callback is used to print out the values of session every time a new session is generated by the tracker:

**iOS:**

```swift
...

let sessionConfig = SessionConfiguration()

    .onSessionStateUpdate { session in

        print("SessionState: id: \(session.sessionId) - index: \(session.sessionIndex) - userID: \(session.userId) - firstEventID: \(session.firstEventId)")

    }

...

let tracker = Snowplow.createTracker(namespace: kNamespace, network: networkConfig, configurations: [sessionConfig])
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration(

    TimeMeasure(6, TimeUnit.SECONDS),

    TimeMeasure(30, TimeUnit.SECONDS)

)

    .onSessionUpdate { state ->

        log(

            "Session: " + state.getSessionId()

                .toString() + "\r\nprevious: " + state.getPreviousSessionId()

                .toString() + "\r\neventId: " + state.getFirstEventId()

                .toString() + "\r\nindex: " + state.getSessionIndex()

                .toString() + "\r\nuserId: " + state.getUserId()

        )

    }



Snowplow.createTracker(applicationContext, namespace, networkConfig, sessionConfig)
```

**Android (Java):**

```java
...

SessionConfiguration sessionConfig = new SessionConfiguration(

    new TimeMeasure(6, TimeUnit.SECONDS),

    new TimeMeasure(30, TimeUnit.SECONDS)

)

    .onSessionUpdate(state -> log(

        "Session: " + state.getSessionId()

                + "\r\nprevious: " + state.getPreviousSessionId()

                + "\r\neventId: " + state.getFirstEventId()

                + "\r\nindex: " + state.getSessionIndex()

                + "\r\nuserId: " + state.getUserId()

    ));



...

Snowplow.createTracker(getApplicationContext(), namespace, networkConfig, sessionConfig);
```

***

## Decorating outgoing links using cross-navigation tracking

> **Note:** This feature was introduced in version 6.0.0 of the iOS and Android trackers.

The `decorateLink` function is part of Snowplow's [cross-navigation solution](/docs/events/cross-navigation/) for tracking the movement of users across different apps and platforms.

Choose which parameters to include using a `CrossDeviceParameterConfiguration` object. The `domainUserId` and `timestamp` are always included automatically. Use booleans to select which properties to include in the link decoration. The tracker will derive the required values from its configuration.

This table shows the options:

| Property         | Description                      | Type    | Included by default? | Value used                         |
| ---------------- | -------------------------------- | ------- | -------------------- | ---------------------------------- |
| `sessionId`      | Current session UUID identifier  | Boolean | ✅                    | `SessionController.sessionId`      |
| `subjectUserId`  | Custom business user identifier  | Boolean | ❌                    | `SubjectController.userId`         |
| `sourceId`       | Application identifier           | Boolean | ✅                    | `TrackerConfiguration.appId`       |
| `sourcePlatform` | Platform of the current device   | Boolean | ❌                    | `TrackerController.devicePlatform` |
| `reason`         | Custom information or identifier | String  | ❌                    | Custom string provided by you      |

The `subjectUserId` property is not included by default, in case it contains personal data.

**iOS:**

```swift
let link = URL(string: "https://example.com")!

let decoratedLink = Snowplow.defaultTracker()?.decorateLink(

  link,

  // optional configuration for which information to be added to the link

  extendedParameters: CrossDeviceParameterConfiguration(sessionId: false, subjectUserId: true)

)
```

**Android (Kotlin):**

```kotlin
val link = Uri.parse("http://example.com")

val decoratedLink = Snowplow.defaultTracker.decorateLink(

  link,

  // optional configuration for which information to be added to the link

  CrossDeviceParameterConfiguration(sessionId = false, subjectUserId = true)

)
```

**Android (Java):**

```java
Uri link = Uri.parse("http://example.com");

Uri decoratedLink = Snowplow.getDefaultTracker().decorateLink(link, new CrossDeviceParameterConfiguration(

    false, // sessionId

    true, // subjectUserId

    false, // sourceId

    false, // sourcePlatform

    null // reason

));
```

***

## Session behavior on app restarts

By default, each time the mobile app restarts (or when a new tracker instance is created), a new session is started regardless of whether the previous session timed out or not.

Since version 6.2 of the iOS and Android trackers, there is an option to continue the previously persisted session when the app restarts. The previous session is only continued in case the app is restarted within the configured session timeout interval. The option can be configured using `SessionConfiguration`:

**iOS:**

```swift
let sessionConfig = SessionConfiguration()

    .continueSessionOnRestart(true)
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration()

    .continueSessionOnRestart(true)
```

**Android (Java):**

```java
SessionConfiguration trackerConfig = new SessionConfiguration()

    .continueSessionOnRestart(true);
```

***

With the option enabled, every session update is persisted to UserDefaults or shared preferences storage. In case it's disabled, only session changes are persisted to UserDefaults. This means that there is more overhead when the option is enabled.

---

# Track visionOS events with the iOS tracker
> Track visionOS-specific events including window groups and immersive spaces in Apple Vision Pro applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/visionos/

> **Note:** Snowplow visionOS tracking was added in version 6.0.0.

The Snowplow iOS tracker supports tracking within visionOS apps. All the usual events, including screen views, can be tracked as for any SwiftUI app. We've also provided additional events and entities to help you understand your visionOS users.

The immersive space context entity is semi-automatically added to all events.

The events are: `OpenWindowEvent`, `DismissWindowEvent`, `OpenImmersiveSpaceEvent`, and `DismissImmersiveSpaceEvent`. Find the schema details for the events and entities [here](/docs/events/ootb-data/visionos-swiftui/).

### Window events

Track the opening and dismissing of SwiftUI window groups using `OpenWindowEvent` and `DismissWindowEvent`. These events can be used in any SwiftUI app, not just visionOS. The event data is sent as a window group context entity (see below) attached to these events; the events themselves have no properties.

#### OpenWindowEvent

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")



let event = OpenWindowEvent(

    id: "group 1", // window group ID

    uuid: UUID(), // optional UUID

    titleKey: "window 1", // optional window title

    windowStyle: WindowStyle.automatic // optional style of the window group

)

tracker.track(event)
```

#### DismissWindowEvent

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")



let event = DismissWindowEvent(

    id: "group 1", // window group ID

    windowId: UUID(), // optional UUID

    titleKey: "window 1", // optional window title

    windowStyle: WindowStyle.automatic // optional style of the window group

)

tracker.track(event)
```

Determining which events occurred in which window group can be done during modelling using these two event types and timestamps. You could also manually add a window group context entity to tracked events.

In this example, a window group entity is added to an [Ecommerce](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) `ProductViewEvent`.

```swift
let product = ProductEntity(

  id: "productId",

  category: "category",

  currency: "GBP",

  price: 100

)

let event = ProductViewEvent(product: product)

let entity = WindowGroupEntity(id: "group_1")

event.entities.append(entity)
```

Read more about customized tracking [here](/docs/sources/mobile-trackers/custom-tracking-using-schemas/).

### Immersive space events

Use the `OpenImmersiveSpaceEvent` and `DismissImmersiveSpaceEvent` to automatically add an immersive space context entity to all events occurring within an immersive space. The entity will identify the immersive space in which the events occurred. This feature is enabled by default.

```swift
let tracker = Snowplow.createTracker(

  namespace: "appTracker",

  endpoint: "https://snowplow-collector-url.com"

)



let event = OpenImmersiveSpaceEvent(

    id: "group 1", // space ID

    viewId: UUID(), // optional UUID

    immersionStyle: ImmersionStyle.automatic, // optional

    upperLimbVisibility: UpperLimbVisibility.hidden // optional

)

// all subsequent events, including this one, will have the immersive space entity

// until DismissImmersiveSpaceEvent is tracked

tracker.track(event)



// the dismiss event will have the entity

// but subsequent events will not

tracker.track(DismissImmersiveSpaceEvent())
```

If the immersive space entity autotracking is turned off, the `Open` and `Dismiss` events can still be tracked. However, only the `OpenImmersiveSpaceEvent` will have the entity. The events themselves have no properties.

The tracker automatically generates a `viewId` each time a new immersive space entity is created. You could also pass in your own UUID.

The visionOS methods `openImmersiveSpace` and `dismissImmersiveSpace` are asynchronous. We advise that you write your tracking code such that the Snowplow immersive space events await those methods' completion, for accurate tracking.

---

# Configure emitters in the .NET tracker
> Configure the emitter for sending and storing events with async processing, persistent storage, and collector endpoints.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/emitter/

These instructions are for Snowplow\.Tracker (.NET Standard) or Snowplow\.Tracker.PlatformExtensions (PCL).

The Emitter object is responsible for sending and storing all events.

We have one emitter available currently:

- `AsyncEmitter` : Fully asynchronous operation which uses threads to perform all of its operations.

The Emitter depends on four other objects being built:

- `IEndpoint`
- `IStorage`
- `IPersistentBlockingQueue`
- `IPayloadToString`

## Emitter Constructor

| **Argument Name**    | **Description**                                                                   | **Required?** | **Default** |
| -------------------- | --------------------------------------------------------------------------------- | ------------- | ----------- |
| `endpoint`           | The endpoint object configured for sending events                                 | Yes           | Null        |
| `queue`              | The queue to be used to push and pop events from                                  | Yes           | Null        |
| `sendLimit`          | The amount of events to get from the queue at a time                              | No            | 100         |
| `stopPollIntervalMs` | The amount of time to wait before checking for more events                        | No            | 300         |
| `sendSuccessMethod`  | An optional callback function which will report event success and failure counts  | No            | Null        |
| `deviceOnlineMethod` | An optional delegate function which will be used to check if the device is online | No            | Null        |
| `logger`             | The logger to use within the application                                          | No            | Null        |

A full Emitter construction should look like the following:

```csharp
var endpoint = new SnowplowHttpCollectorEndpoint(emitterUri, method: method, port: port, protocol: protocol, l: logger);

var storage = new LiteDBStorage("events.db");

var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());



AsyncEmitter emitter = new AsyncEmitter(endpoint, queue, l: logger);
```

**NOTE**: The send limit can impact performance greatly as it determines how often we need to perform I/O to the disk and how big the POSTed event batches can be.

**WARNING**: If you are sending events via GET note that each event is sent as its own task, so this has the potential to launch 100 outbound tasks in parallel. It is recommended to lower this range if using GET to 10-15 as a maximum.

## Endpoint Constructor

This is a container for information about how to reach your collector.

| **Argument Name** | **Description**                                        | **Required?** | **Default**       |
| ----------------- | ------------------------------------------------------ | ------------- | ----------------- |
| `host`            | The collector uri to send events to                    | Yes           | Null              |
| `protocol`        | The protocol to use when sending events (HTTP / HTTPs) | No            | HttpProtocol.HTTP |
| `port`            | If the collector is not on port 80                     | No            | Null              |
| `method`          | The method to use when sending (GET / POST)            | No            | HttpMethod.GET    |
| `postMethod`      | Custom method for sending events via POST              | No            | Null              |
| `getMethod`       | Custom method for sending events via GET               | No            | Null              |
| `byteLimitPost`   | Maximum byte limit when sending a POST request         | No            | 40000             |
| `byteLimitGet`    | Maximum byte limit when sending a GET request          | No            | 40000             |
| `logger`          | The logger to use within the application               | No            | Null              |

We have one endpoint available currently:

- `SnowplowHttpCollectorEndpoint`

A full Endpoint construction should look like the following:

```csharp
SnowplowHttpCollectorEndpoint endpoint = new SnowplowHttpCollectorEndpoint("com.acme-collector", protocol: HttpProtocol.HTTPS, method: HttpMethod.GET, l: logger);
```

**NOTE**: If any individual event exceeds the byte limits set then this event will be sent - but it will be assumed to have succeeded. This is to prevent constanstly attempting to send overly large events.

## Storage Constructor

| **Argument Name** | **Description**                             | **Required?** | **Default** |
| ----------------- | ------------------------------------------- | ------------- | ----------- |
| `path`            | The file path to store the database file at | Yes           | Null        |

We have one storage target available currently:

- `LiteDBStorage`

A full Storage construction should look like the following:

```csharp
LiteDBStorage storage = new LiteDBStorage("events.db");
```

**NOTE**: When using the Tracker within Xamarin you will need to fetch a correct path for internal storage. Some example code for fetching this path:

```csharp
// Android

public string GetLocalFilePath(string filename)

{

  string path = Environment.GetFolderPath(Environment.SpecialFolder.Personal);

  return Path.Combine(path, filename);

}



// iOS

public string GetLocalFilePath(string filename)

{

  string docFolder = Environment.GetFolderPath(Environment.SpecialFolder.Personal);

  string libFolder = Path.Combine(docFolder, "..", "Library", "Databases");



  if (!Directory.Exists(libFolder))

  {

    Directory.CreateDirectory(libFolder);

  }



  return Path.Combine(libFolder, filename);

}
```

## Queue Constructor

| **Argument Name** | **Description**                          | **Required?** | **Default** |
| ----------------- | ---------------------------------------- | ------------- | ----------- |
| `storage`         | The storage object to use with the queue | Yes           | Null        |
| `payloadToString` | Serializer for Payload objects           | Yes           | Null        |

We have one queue available currently:

- `PersistentBlockingQueue`

A full queue construction should look like the following:

```csharp
PersistentBlockingQueue queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());
```

## Payload Serializer Constructor

We have one payload serializer available currently:

- `PayloadToJsonString`

A full queue construction should look like the following:

```csharp
PayloadToJsonString serializer = new PayloadToJsonString();
```

This controls how we queue information for internal use.

---

# Track events with the .NET tracker
> Track page views, structured events, screen views, and custom self-describing events with the .NET tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/event-tracking/

Events supported by the .NET Tracker at a glance:

| **Events**                    | **Description**                                        |
| ----------------------------- | ------------------------------------------------------ |
| `Track(PageView)`             | Track and record views of web pages                    |
| `Track(ScreenView)`           | Track the user viewing a screen within the application |
| `Track(Structured)`           | Track a Snowplow custom structured event               |
| `Track(Timing)`               | Track a Timing with Category event                     |
| `Track(Unstructured)`         | Track a Snowplow custom unstructured event             |
| `Track(EcommerceTransaction)` | Track an ecommerce transaction and its items           |

### Track page views with `Track(PageView)`

You can use `Track(PageView)` to track a user viewing a web page within your app.

Arguments are:

| **Argument**     | **Description**                      | **Required?** | **Type**         |
| ---------------- | ------------------------------------ | ------------- | ---------------- |
| `pageUrl`        | The URL of the page                  | Yes           | `string`         |
| `pageTitle`      | The title of the page                | No            | `string`         |
| `referrer`       | The address which linked to the page | No            | `string`         |
| `customContexts` | Optional custom context              | No            | `List<IContext>` |
| `trueTimestamp`  | Optional true-timestamp              | No            | `long`           |
| `eventId`        | Optional custom event id             | No            | `string`         |

Examples:

```csharp
t1.Track(new PageView()

    .SetPageUrl("www.example.com")

    .SetPageTitle("example")

    .SetReferrer("www.referrer.com")

    .Build());



t1.Track(new PageView()

    .SetPageUrl("www.example.com")

    .SetPageTitle("example")

    .SetReferrer("www.referrer.com")

    .SetCustomContext(contextList)

    .SetTrueTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

### Track screen views with `Track(ScreenView)`

Use `Track(ScreenView)` to track a user viewing a screen (or equivalent) within your app. You **must** use either `name` or `id`. Arguments are:

| **Argument**             | **Description**                     | **Required?** | **Type**         |
| ------------------------ | ----------------------------------- | ------------- | ---------------- |
| `name`                   | Human-readable name for this screen | No            | `string`         |
| `id`                     | Unique identifier for this screen   | No            | `string`         |
| `customContexts`         | Optional custom context             | No            | `List<IContext>` |
| `deviceCreatedTimestamp` | Optional device-created-timestamp   | No            | `long`           |
| `trueTimestamp`          | Optional true-timestamp             | No            | `long`           |
| `eventId`                | Optional custom event id            | No            | `string`         |

Examples:

```csharp
t1.Track(new ScreenView()

    .SetName("HUD > Save Game")

    .SetId("screen23")

    .Build());



t1.Track(new ScreenView()

    .SetName("HUD > Save Game")

    .SetId("screen23")

    .SetCustomContext(contextList)

    .SetTrueTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

### Track structured events with `Track(Structured)`

Use `Track(Structured)` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument**             | **Description**                                                  | **Required?** | **Type**         |
| ------------------------ | ---------------------------------------------------------------- | ------------- | ---------------- |
| `category`               | The grouping of structured events which this `action` belongs to | Yes           | `string`         |
| `action`                 | Defines the type of user interaction which this event involves   | Yes           | `string`         |
| `label`                  | A string to provide additional dimensions to the event data      | No            | `string`         |
| `property`               | A string describing the object or the action performed on it     | No            | `string`         |
| `value`                  | A value to provide numerical data about the event                | No            | `double`         |
| `customContexts`         | Optional custom context                                          | No            | `List<IContext>` |
| `deviceCreatedTimestamp` | Optional device-created-timestamp                                | No            | `long`           |
| `trueTimestamp`          | Optional true-timestamp                                          | No            | `long`           |
| `eventId`                | Optional custom event id                                         | No            | `string`         |

Examples:

```csharp
t1.Track(new Structured()

    .SetCategory("shop")

    .SetAction("add-to-basket")

    .Build());



t1.Track(new Structured()

    .SetCategory("shop")

    .SetAction("add-to-basket")

    .SetLabel("Add To Basket")

    .SetProperty("pcs")

    .SetValue(2.00)

    .SetCustomContext(contextList)

    .SetTrueTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

### Track timing events with `Track(Timing)`

Use `Track(Timing)` to track an event related to a custom timing.

| **Argument**             | **Description**                        | **Required?** | **Type**         |
| ------------------------ | -------------------------------------- | ------------- | ---------------- |
| `category`               | The category of the timed event        | Yes           | `string`         |
| `label`                  | The label of the timed event           | No            | `string`         |
| `timing`                 | The timing measurement in milliseconds | Yes           | `int`            |
| `variable`               | The name of the timed event            | Yes           | `string`         |
| `customContexts`         | Optional custom context                | No            | `List<IContext>` |
| `deviceCreatedTimestamp` | Optional device-created-timestamp      | No            | `long`           |
| `trueTimestamp`          | Optional true-timestamp                | No            | `long`           |
| `eventId`                | Optional custom event id               | No            | `string`         |

Examples:

```csharp
t1.Track(new Timing()

    .SetCategory("category")

    .SetVariable("variable")

    .SetTiming(1)

    .Build());



t1.Track(new Timing()

    .SetCategory("category")

    .SetVariable("variable")

    .SetTiming(1)

    .SetLabel("label")

    .SetCustomContext(contextList)

    .SetTrueTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

#### Track unstructured events with `Track(SelfDescribing)`

Custom unstructured events are a flexible tool that enable Snowplow users to define their own event types and send them into Snowplow.

When a user sends in a custom unstructured event, they do so as a JSON of name-value properties, that conforms to a JSON schema defined for the event earlier.

Use `Track(SelfDescribing)` to track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

The arguments are as follows:

| **Argument**             | **Description**                   | **Required?** | **Type**             |
| ------------------------ | --------------------------------- | ------------- | -------------------- |
| `eventData`              | The properties of the event       | Yes           | `SelfDescribingJson` |
| `customContexts`         | Optional custom context           | No            | `List<IContext>`     |
| `deviceCreatedTimestamp` | Optional device-created-timestamp | No            | `long`               |
| `trueTimestamp`          | Optional true-timestamp           | No            | `long`               |
| `eventId`                | Optional custom event id          | No            | `string`             |

Example event json to track:

```csharp
{

  "schema": "iglu:com.acme/save_game/jsonschema/1-0-0",

  "data": {

    "levelName": "Barrels o' Fun",

    "levelIndex": 23

  }

}
```

How to set it up?

```csharp
// Create a Dictionary of your event data

Dictionary<string, object> eventDict = new Dictionary<string, object>();

eventDict.Add("levelName", "Barrels o' Fun");

eventDict.Add("levelIndex", 23);



// Create your event data

SelfDescribingJson eventData = new SelfDescribingJson("iglu:com.acme/save_game/jsonschema/1-0-0", eventDict);



// Track your event with your custom event data

t1.Track(new SelfDescribing()

    .SetEventData(eventData)

    .Build());



// OR



t1.Track(new SelfDescribing()

    .SetEventData(eventData)

    .SetCustomContext(contextList)

    .SetTrueTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

For more on JSON schema, see the [blog post](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

### Track ecommerce transactions with `Track(EcommerceTransaction)`

Use `Track(EcommerceTransaction)` to track an ecommerce transaction.

Arguments:

| **Argument**             | **Description**                   | **Required?** | **Type**                         |
| ------------------------ | --------------------------------- | ------------- | -------------------------------- |
| `orderId`                | ID of the eCommerce transaction   | Yes           | `string`                         |
| `totalValue`             | Total transaction value           | Yes           | `double`                         |
| `affiliation`            | Transaction affiliation           | No            | `string`                         |
| `taxValue`               | Transaction tax value             | No            | `double`                         |
| `shipping`               | Delivery cost charged             | No            | `double`                         |
| `city`                   | Delivery address city             | No            | `string`                         |
| `state`                  | Delivery address state            | No            | `string`                         |
| `country`                | Delivery address country          | No            | `string`                         |
| `currency`               | Transaction currency              | No            | `string`                         |
| `items`                  | Items in the transaction          | Yes           | `List<EcommerceTransactionItem>` |
| `customContexts`         | Optional custom context           | No            | `List<IContext>`                 |
| `deviceCreatedTimestamp` | Optional device-created-timestamp | No            | `long`                           |
| `trueTimestamp`          | Optional true-timestamp           | No            | `long`                           |
| `eventId`                | Optional custom event id          | No            | `string`                         |

The `items` argument is a `List` of individual `EcommerceTransactionItem` elements representing the items in the ecommerce transaction. Note that `Track(EcommerceTransaction)` fires multiple events: one transaction event for the transaction as a whole, and one transaction item event for each element of the `items` `List`.

Each transaction item event will have the same device created timestamp, true timestamp, orderId, and currency as the main transaction event.

#### `EcommerceTransactionItem`

#### Using Snowplow\.Tracker (.NET Standard) or Snowplow\.Tracker.PlatformExtensions (PCL)

To instantiate a `EcommerceTransactionItem` in your code, simply use the following constructor signature:

```csharp
EcommerceTransactionItem item = new EcommerceTransactionItem ()

      .SetSku ("sku")

      .SetPrice (10.2)

      .SetQuantity (1)

      .SetName ("name")

      .SetCategory ("category")

      .Build ()
```

These are the fields that can appear as elements in each `EcommerceTransactionItem` element of the transaction item's `List`:

| **Field**        | **Description**          | **Required?** | **Type**         |
| ---------------- | ------------------------ | ------------- | ---------------- |
| `sku`            | Item SKU                 | Yes           | `string`         |
| `price`          | Item price               | Yes           | `double`         |
| `quantity`       | Item quantity            | Yes           | `int`            |
| `name`           | Item name                | No            | `string`         |
| `category`       | Item category            | No            | `string`         |
| `customContexts` | Optional custom context  | No            | `List<IContext>` |
| `eventId`        | Optional custom event id | No            | `string`         |

Example of tracking a transaction containing two items:

```csharp
// Create some Transaction Items

EcommerceTransactionItem item1 = new EcommerceTransactionItem ()

    .SetSku ("item_sku_1")

    .SetPrice (10.2)

    .SetQuantity (1)

    .SetName ("item_name_1")

    .SetCategory ("item_category")

    .Build ();



EcommerceTransactionItem item2 = new EcommerceTransactionItem()

    .SetSku("item_sku_2")

    .SetPrice(1.00)

    .SetQuantity(1)

    .SetName("item_name_2")

    .SetCategory("item_category")

    .Build();



// Add these items to a List

List<EcommerceTransactionItem> items = new List<EcommerceTransactionItem>();

items.Add(item1);

items.Add(item2);



// Now Track the Transaction by using this list of items as an argument

tracker.Track(new EcommerceTransaction()

    .SetOrderId("order_id_1")

    .SetTotalValue(300.00)

    .SetAffiliation("my_affiliate")

    .SetTaxValue(30.00)

    .SetShipping(10.00)

    .SetCity("Boston")

    .SetState("Massachusetts")

    .SetCountry("USA")

    .SetCurrency("USD")

    .SetItems(items)

    .Build());
```

### Custom Contexts

Custom contexts are Self Describing Jsons with extra descriptive information that can be optionally attached to any Snowplow event with `SetCustomContexts(...)`. We provide several builders for Snowplow custom contexts as well as a generic builder if you wish to define and send your own custom contexts!

For ease of development you are also able to extend the `IContext` interface or the `AbstractContext` class for your own contexts if you so wish.

All of these contexts will need to be combined into a `List<IContext>` before being attachable to Snowplow Events.

#### `DesktopContext`

The following arguments can be used in a DesktopContext:

| **Field**            | **Description**                     | **Required?** | **Type** |
| -------------------- | ----------------------------------- | ------------- | -------- |
| `osType`             | The Operating System Type           | Yes           | `string` |
| `osVersion`          | The Version of the Operating System | Yes           | `string` |
| `osServicePack`      | Service Pack information            | No            | `string` |
| `osIs64Bit`          | If the OS is 32 or 64 bit           | No            | `bool`   |
| `deviceManufacturer` | Who made the device                 | No            | `string` |
| `deviceModel`        | What is the device model            | No            | `string` |
| `processorCount`     | How many cores does the device have | No            | `int`    |

An example of a DesktopContext construction:

```csharp
DesktopContext context = new DesktopContext ()

    .SetOsType("OS-X")

    .SetOsVersion("10.10.5")

    .SetOsServicePack("Yosemite")

    .SetOsIs64Bit(true)

    .SetDeviceManufacturer("Apple")

    .SetDeviceModel("Macbook Pro")

    .SetDeviceProcessorCount(4)

    .Build ();
```

#### `MobileContext`

The following arguments can be used in a MobileContext:

| **Field**            | **Description**                     | **Required?** | **Type**      |
| -------------------- | ----------------------------------- | ------------- | ------------- |
| `osType`             | The Operating System Type           | Yes           | `string`      |
| `osVersion`          | The Version of the Operating System | Yes           | `string`      |
| `deviceManufacturer` | Who made the device                 | Yes           | `string`      |
| `deviceModel`        | What is the device model            | Yes           | `string`      |
| `carrier`            | The name of the carrier             | No            | `string`      |
| `networkType`        | The type of network                 | No            | `NetworkType` |
| `networkTechnology`  | The networks technlogy              | No            | `string`      |
| `openIdfa`           | An OpenIDFA UUID                    | No            | `string`      |
| `appleIdfa`          | An Apple IDFA UUID                  | No            | `string`      |
| `appleIdfv`          | An Apple IDFV UUID                  | No            | `string`      |
| `androidIdfa`        | An Android IDFA UUID                | No            | `string`      |

An example of a MobileContext construction:

#### `GeoLocationContext`

The following arguments can be used in a GeoLocationContext:

| **Field**                   | **Description**            | **Required?** | **Type** |
| --------------------------- | -------------------------- | ------------- | -------- |
| `latitude`                  | The user latitude          | Yes           | `double` |
| `longitude`                 | The user longitude         | Yes           | `double` |
| `latitudeLongitudeAccuracy` | The user lat-long accuracy | No            | `double` |
| `altitude`                  | The user altitude          | No            | `double` |
| `altitudeAccuracy`          | The user alt accuracy      | No            | `double` |
| `bearing`                   | The user bearing           | No            | `double` |
| `speed`                     | The user speed             | No            | `double` |
| `timestamp`                 | A timestamp in ms          | No            | `long`   |

An example of a GeoLocationContext construction:

```csharp
GeoLocationContext context = new GeoLocationContext ()

    .SetLatitude(123.564)

    .SetLongitude(-12.6)

    .SetLatitudeLongitudeAccuracy(5.6)

    .SetAltitude(5.5)

    .SetAltitudeAccuracy(2.1)

    .SetBearing(3.2)

    .SetSpeed(100.2)

    .SetTimestamp(1234567890000)

    .Build ();
```

#### `GenericContext`

The GenericContext is a simple builder with three functions:

- `SetSchema(string)` : Sets the Context Schema Path
- `Add(string, object)` : Adds a single key-pair value to the data packet of this context
- `AddDict(string, object)` : Adds a dictionary of key-pair values to the data packet

You must set a schema string or a RuntimeException will be thrown.

An example of a GenericContext construction:

```csharp
GenericContext context = new GenericContext()

    .SetSchema("iglu:com.acme/acme_context/jsonschema/1-0-0")

    .Add("context", "custom")

    .Build();
```

### SelfDescribingJson

A `SelfDescribingJson` is used as a wrapper around a `Dictionary<string, object>`. After creating the Dictionary you want to wrap you can create a `SelfDescribingJson` using the following:

```csharp
// Data as a Dictionary

Dictionary<string, object> data = new Dictionary<string, object>();

data.Add("Event", "Data")



// We then create a new SelfDescribingJson

SelfDescribingJson json = new SelfDescribingJson("iglu:com.acme/example/jsonschema/1-0-0", data);
```

This object is now ready to be Tracked within an SelfDescribing Event.

You can create a SelfDescribingJson with the following arguments:

| **Argument** | **Description**                           | **Required?** | **Type**                    |
| ------------ | ----------------------------------------- | ------------- | --------------------------- |
| `schema`     | JsonSchema that describes the data        | Yes           | `string`                    |
| `data`       | Data that will be validated by the schema | No            | `Dictionary<string,object`> |

---

# .NET tracker SDK
> Track events in .NET websites and desktop applications with the Snowplow .NET tracker SDK for .NET Standard and Xamarin.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/

The Snowplow .NET Tracker allows you to track Snowplow events from your .NET websites and desktop applications. It's split into two libraries:

#### `Snowplow.Tracker`

This is a fully functional .NET Standard 1.4 tracking library - it will work on any platform that supports .NET Standard 1.4+ (including .NET 461+).

#### `Snowplow.Tracker.PlatformExtensions`

This is a Portable Class Library (PCL) wrapper around the core `Snowplow.Tracker` library that extends functionality in platform specific ways, for example to provide geo-location information when tracking users in a Xamarin mobile application. If you're using Xamarin we encourage you to use this library.

## Snowplow Demo Application

The Xamarin demo application can be deployed on Android and iOS. Simply launch the `Snowplow.Demo.App.sln` solution file with Visual Studio and deploy to either emulators or actual test devices. The .NET Core Console demo application can also be loaded with Visual Studio using `Snowplow.Demo.Console.sln`.

---

# Initialize the .NET tracker
> Initialize and configure the .NET tracker with emitter, storage, and subject settings for event tracking.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/initialization/

Assuming you have completed the .NET Tracker Setup for your project, you are now ready to initialize the .NET Tracker.

### Importing the library

Add the following `using` lines to the top of your `.cs` scripts to access the Tracker:

```csharp
using Snowplow.Tracker.Emitters;

using Snowplow.Tracker.Endpoints;

using Snowplow.Tracker.Logging;

using Snowplow.Tracker.Models;

using Snowplow.Tracker.Models.Events;

using Snowplow.Tracker.Models.Adapters;

using Snowplow.Tracker.Queues;

using Snowplow.Tracker.Storage;
```

You should now be able to setup the Tracker!

### Creating a tracker

#### Using Snowplow\.Tracker (.NET Standard) or Snowplow\.Tracker.PlatformExtensions (PCL)

To use the Tracker in your code simply instantiate the Tracker interface with the following:

```csharp
// Create logger

var logger = new ConsoleLogger();



// Controls the sending of events

var endpoint = new SnowplowHttpCollectorEndpoint(emitterUri, method: method, port: port, protocol: protocol, l: logger);



// Controls the storage of events

// NOTE: You must dispose of storage yourself when closing your application!

var storage = new LiteDBStorage("events.db");



// Controls queueing events

var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());



// Controls pulling events of the queue and pushing them to the sender

var emitter = new AsyncEmitter(endpoint, queue, l: logger);



// Contains information about who you are tracking

var subject = new Subject().SetPlatform(Platform.Mob).SetLang("EN");



Tracker.Tracker.Instance.Start(emitter: emitter, subject: subject, trackerNamespace: "some namespace", appId: "some appid", l: logger);
```

This starts a global singleton Tracker which can be accessed anywhere via the `Tracker.Tracker.Instance.{{ method }}` chain.

---

# Platform specific functions for the .NET tracker
> Access platform-specific information for mobile, desktop, and geo-location contexts in Xamarin and .NET applications.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/platform-specific-functions/

To support multiple platforms we provide several utility functions for fetching platform specific information.

### `GetMobileContext`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`

**NOTE**: These mobile contexts do not currently fetch any advertising identifiers.

### `GetGeoLocationContext`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`

**NOTE**: To make the `GeoLocation` context work on `iOS` you will need to add the following to your `Info.plist`:

```csharp
<key>UIBackgroundModes</key>

<array>

  <string>location</string>

</array>

<key>NSLocationAlwaysUsageDescription</key>

<string></string>
```

When you send your first event a prompt will be fired asking the user for permission to use their location.

### `GetDesktopContext`

#### PlatformExtensions only

Platforms:

- `.NET Framework 4.6.1+`

### 8.4 `GetLocalFilePath`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`

### 8.5 `IsDeviceOnline`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`
- `.NET Framework 4.6.1+`

---

# Track session data with the .NET tracker
> Track user sessions with persistent session data including session ID, index, and timeout configuration for .NET applications.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/session/

The Session object is responsible for maintaining persistent data around user sessions over the life-time of an application.

### Constructor

| **Argument Name**   | **Description**                           | **Required?** | **Default** |
| ------------------- | ----------------------------------------- | ------------- | ----------- |
| `savePath`          | The path to save persistent data into     | Yes           | Null        |
| `foregroundTimeout` | The time until a session expires in focus | No            | 600 (s)     |
| `backgroundTimeout` | The time until a session expires in back  | No            | 300 (s)     |
| `checkInterval`     | How often to validate the session timeout | No            | 15 (s)      |
| `logger`            | The logger to use within the application  | No            | Null        |

A full ClientSession construction should look like the following:

```csharp
ClientSession session = new ClientSession ("save_file_path.xml");
```

The timeout's refer to the length of time the session remains active after the last event is sent. As long as events are sent within this limit the session will not timeout.

**NOTE**: When using the Tracker within Xamarin you will need to fetch a correct path for internal storage. Some example code for fetching this path:

```csharp
// Android

public string GetLocalFilePath(string filename)

{

  string path = Environment.GetFolderPath(Environment.SpecialFolder.Personal);

  return Path.Combine(path, filename);

}



// iOS

public string GetLocalFilePath(string filename)

{

  string docFolder = Environment.GetFolderPath(Environment.SpecialFolder.Personal);

  string libFolder = Path.Combine(docFolder, "..", "Library", "Databases");



  if (!Directory.Exists(libFolder))

  {

    Directory.CreateDirectory(libFolder);

  }



  return Path.Combine(libFolder, filename);

}
```

### 6.2 Functions

#### `SetBackground(bool)`

Will set whether or not the application is in the background. It is up to the developer to set this metric if they wish to have a different timeout for foreground and background.

**NOTE**: If a timeout occurs while the application has been backgrounded the SessionChecker will be stopped automatically. Session checking will resume on Foregrounding or on an event being tracked.

---

# Install the .NET tracker
> Install the .NET tracker using NuGet for .NET Standard, Xamarin, and MAUI applications.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/setup/

Version 1.0.0+ marks the conversion of the core library to .NET Standard 1.4. We have also published a Portable Class Library (PCL) wrapper providing extra features for Xamarin users. This is called `Snowplow.Tracker.PlatformExtensions`.

The current version is 1.3.0.

## Setup

### .NET Standard users

To add the .NET Tracker as a dependency to your project, install it in the Visual Studio Package Manager Console using [NuGet](https://www.nuget.org/):

```powershell
Install-Package Snowplow.Tracker
```

### Xamarin users

To add the .NET Tracker as a dependency to your project, install it in the Visual Studio Package Manager Console using [NuGet](https://www.nuget.org/):

```powershell
Install-Package Snowplow.Tracker.PlatformExtensions
```

Remember to add an assembly reference to the .NET Tracker to your project.

### MAUI users

The `Snowplow.Tracker.PlatformExtensions` package is currently not compatible with MAUI apps.

However, as a replacement for the `GetLocalFilePath` function that the package provides, you can use the following implementation:

```cs
/// <summary>

/// Returns a path for storing the Snowplow events and session databases

/// </summary>

public string GetLocalFilePath(string filename)

{

    var path = System.Environment.GetFolderPath(System.Environment.SpecialFolder.LocalApplicationData);

    return Path.Combine(path, filename);

}
```

This function can be used to initialize the tracker as follows:

```cs
var dest = new SnowplowHttpCollectorEndpoint(collectorUri, method: method, port: port, protocol: protocol);

var storage = new LiteDBStorage(GetLocalFilePath("events.db"));

var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());

var emitter = new AsyncEmitter(dest, queue);

var clientSession = new ClientSession(GetLocalFilePath("client_session.dict"));

Instance.Start(emitter: emitter, clientSession: clientSession);
```

---

# Configure a subject with the .NET tracker
> Add user and environment data to tracked events using the Subject class with platform, language, and user properties.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/subject/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

The Subject class has a number of `Set...()` methods to attach extra data relating to the user of the tracked events.

Here are some examples:

```csharp
Subject s1 = new Subject();

s1.SetUserId("Kevin Gleason");

s1.SetLang("en-gb");

s1.SetScreenResolution(1920, 1080);
```

There are two ways to provide the subject information:

1. Globally for all tracked events. This is useful in client-side applications where all tracked events relate to the same user and share the same properties.
2. For each event individually. This is useful in server-side applications where each event might relate to a different user.

To configure a global subject, pass it to the tracker during initialization:

```csharp
var globalSubject = new Subject().SetPlatform(Platform.Mob).SetLang("EN");



Tracker.Tracker.Instance.Start(emitter: emitter, subject: globalSubject, trackerNamespace: "some namespace", appId: "some appid", l: logger);
```

To pass the subject along with tracked events:

```csharp
Subject eventSubject = new Subject().SetUserId("Kevin Gleason");



Tracker.Instance.Track(

    new Structured()

        .SetCategory("shop")

        .SetAction("add-to-basket")

        .Build(),

    eventSubject

);
```

### `SetUserId`

You can set the user ID to any string:

```csharp
s1.SetUserId( "{{USER ID}}" )
```

Example:

```csharp
s1.SetUserId("alexd")
```

### `SetScreenResolution`

If your C# code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```csharp
s1.SetScreenResolution( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
s1.SetScreenResolution(1366, 768)
```

### `SetViewport`

If your C# code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```csharp
s1.SetViewport( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
s1.SetViewport(300, 200)
```

### `SetColorDepth`

If your C# code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```csharp
s.SetColorDepth( {{BITS PER PIXEL}} )
```

The number should be a positive integer, in bits per pixel. Example:

```csharp
s.SetColorDepth(32)
```

### `SetTimezone`

This method lets you pass a user's timezone in to Snowplow:

```csharp
s.SetTimezone( {{TIMEZONE}} )
```

The timezone should be a string:

```csharp
s.SetTimezone("Europe/London")
```

### `SetLang`

This method lets you pass a user's language in to Snowplow:

```csharp
s.SetLang( {{LANGUAGE}} )
```

The language should be a string:

```csharp
s.SetLang('en')
```

### `SetIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```csharp
s.SetIpAddress( {{IP ADDRESS}} )
```

The IP address should be a string:

```csharp
s.SetIpAddress("127.0.0.1");
```

### `SetUseragent`

This method lets you pass a useragent in to Snowplow:

```csharp
s.SetUseragent( {{USERAGENT}} )
```

The useragent should be a string:

```csharp
s.SetUseragent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0");
```

### `SetNetworkUserId`

This method lets you pass a Network User ID in to Snowplow:

```csharp
s.SetNetworkUserId( {{NUID}} )
```

The network user id should be a string, typically a UUID:

```csharp
s.SetNetworkUserId("a57c0cac-4aab-49c5-8e70-790f415080b0");
```

### `SetDomainUserId`

This method lets you pass a Domain User ID in to Snowplow:

```csharp
s.SetDomainUserId( {{DUID}} )
```

The domain user id should be a string, typically a UUID:

```csharp
s.SetDomainUserId("be0999db-8e4e-4f0c-9f3e-fe5e5de1669a");
```

### `SetDomainSessionId`

This method lets you pass a Domain Session ID in to Snowplow:

```csharp
s.SetDomainSessionId( {{SID}} )
```

The domain session id should be a string, typically a UUID:

```csharp
s.SetDomainSessionId("8b107139-5b38-45cc-a0eb-be2550e32904");
```

### `SetDomainSessionIndex`

This method lets you pass a Domain Session Index in to Snowplow:

```csharp
s.SetDomainSessionIndex( {{VID}} )
```

The domain user id should be an int:

```csharp
s.SetDomainSessionIndex(1);
```

### `SetPlatform`

This method lets you set the Platform that the Tracker is running on:

```csharp
s.SetPlatform( Platform.{{ option }} )
```

The Platform should be an enum:

```csharp
s.SetPlatform(Platform.Mob);
```

Available platforms:

- Web
- Mob
- Pc
- Srv
- App
- Tv
- Cnsl
- Iot

---

# Configure a tracker with the .NET tracker
> Configure the .NET tracker instance with emitter, subject, session, and logging options for coordinating event tracking.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/tracker/

The Tracker object is responsible for co-ordinating the saving and sending of events as well as managing the optional Session object.

## Constructor

| **Argument Name**            | **Description**                                                          | **Required?** | **Default** |
| ---------------------------- | ------------------------------------------------------------------------ | ------------- | ----------- |
| `emitter`                    | The Emitter object you create                                            | Yes           | Null        |
| `subject`                    | The Subject that defines a user                                          | No            | Null        |
| `clientSession`              | The Session object you create                                            | No            | Null        |
| `trackerNamespace`           | The name of the tracker instance                                         | No            | Null        |
| `appId`                      | The application ID                                                       | No            | Null        |
| `base64Encoded`              | If we [base 64 encode](https://en.wikipedia.org/wiki/Base64) json values | No            | True        |
| `synchronous`                | If loading into storage is done in sync                                  | No            | True        |
| `desktopContextDelegate`     | Function to get the desktop context                                      | No            | Null        |
| `mobileContextDelegate`      | Function to get the mobile context                                       | No            | Null        |
| `geoLocationContextDelegate` | Function to get the geo-location context                                 | No            | Null        |
| `logger`                     | The logger to use within the application                                 | No            | Null        |

A full Tracker construction should look like the following:

```csharp
var logger = new ConsoleLogger();

var endpoint = new SnowplowHttpCollectorEndpoint(emitterUri, method: method, port: port, protocol: protocol, l: logger);

var storage = new LiteDBStorage("events.db");

var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());

var emitter = new AsyncEmitter(endpoint, queue, l: logger);

var subject = new Subject().SetPlatform(Platform.Mob).SetLang("EN");

var session = new ClientSession("client_session.dict", l: logger);



Tracker.Instance.Start(emitter: emitter, subject: subject, clientSession: session, trackerNamespace: "some namespace", appId: "some appid", encodeBase64: true, l: logger);
```

The `Tracker.Start(...)` and `Tracker.Stop()` methods take full responsibility for starting and stopping the threads required for processing everything asynchronously. **Do not** use any other `Start` and `Stop` functions other than the ones directly for the Tracker to prevent unknown behaviors.

**WARNING**: The `LiteDBStorage` object must be disposed of manually after stopping the Tracker so you will need to maintain a reference to this object.

**NOTE**: The `Subject` variables can all be altered directly from the Tracker via replicated setter methods.

## Functions

The Tracker contains several critical functions that must be used to start Tracking.

### `Start(...)`

This function must be called before any events will start being stored or sent. This is due to the fact that we do not want to start any background processing from the constructors so it is left up to the developer to choose when to start everything up.

If you attempt to access the Tracker singleton before Starting it an exception will be thrown.

This function:

- Starts the background emitter thread
- Starts the background session check timer (Optional)

Once this is run everything should be in place for asynchronous event tracking.

### `Stop()`

If you need to halt the Tracker from tracking events you can run this function. This will bring all event processing, sending and collection to a halt and nothing will be started again until `Start(...)` is fired again.

**WARNING**: If you are using Client Sessionization stopping and then restarting the Tracker will result in the session index incrementing.

### `Track(IEvent, Subject)`

This is the function used for Tracking all events.

```csharp
Tracker.Instance.Track(IEvent newEvent, Subject eventSubject = null);
```

---

# Configure the Node.js tracker
> Configure the Node.js tracker with platform, user ID, session properties, and device information for enriched event tracking.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/configuration/

You may have additional information about your application's environment, current user and so on, which you want to send to your Snowplow pipeline with each event.

#### Setting event properties

The `tracker` instance has a set of `set...()` methods to attach extra data to all tracked events:

- [Set the platform ID with `setPlatform()`](#set-the-platform-id-withsetplatform)
- [Set user ID with `setUserId()`](#set-user-id-withsetuserid)
- [Set domain user ID with `setDomainUserId()`](#set-domain-user-id-with-setdomainuserid)
- [Set session ID with `setSessionId()`](#set-session-id-with-setsessionid)
- [Set session index with `setSessionIndex()`](#set-session-index-with-setsessionindex)
- [Set network user ID with `setNetworkUserId()`](#set-network-user-id-with-setnetworkuserid)
- [Set screen resolution with `setScreenResolution()`](#set-screen-resolution-with-setscreenresolution)
- [Set viewport dimensions with `setViewport()`](#set-viewport-dimensions-withsetviewport)
- [Set color depth with `setColorDepth()`](#set-color-depth-withsetcolordepth)
- [Set the timezone with `setTimezone()`](#set-the-timezone-withsettimezone)
- [Set the language with `setLang()`](#set-the-language-withsetlang)
- [Set the IP Address with `setIpAddress()`](#set-the-ip-address-withsetipaddress)
- [Set the Useragent with `setUseragent()`](#set-the-useragent-withsetuseragent)

### Set the platform ID with `setPlatform()`

You can set the platform:

```javascript
t.setPlatform('{{PLATFORM}}');
```

Example:

```javascript
t.setPlatform('mob');
```

If the platform is not set manually, it defaults to `'srv'` (for server).

For a full list of supported platforms, please see the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields).

### Set user ID with `setUserId()`

You can set the user ID to any string:

```javascript
t.setUserId('{{USER ID}}');
```

Example:

```javascript
t.setUserId('alexd');
```

### Set domain user ID with `setDomainUserId()`

You can set the domain user ID to any string:

```javascript
t.setDomainUserId('{{DOMAIN USER ID}}');
```

Example:

```javascript
t.setDomainUserId('e8091074-f197-4279-a92e-71a34171d477');
```

### Set session ID with `setSessionId()`

You can set the session ID to any string:

```javascript
t.setSessionId('{{SESSION ID}}');
```

Example:

```javascript
t.setSessionId('s8091074-f197-4279-a92e-71a34171d477');
```

### Set session index with `setSessionIndex()`

You can set the session index to any number:

```javascript
t.setSessionIndex('{{SESSION INDEX}}');
```

Example:

```javascript
t.setSessionIndex(10);

/* t.setSessionIndex('10'); will also work as long as input string can be converted to a valid number. */
```

### Set network user ID with `setNetworkUserId()`

You can set the network user ID to any string:

```javascript
t.setNetworkUserId('{{NETWORK USER ID}}');
```

Example:

```javascript
t.setNetworkUserId('b98c76ad-e45c-45db-a101-224718064ac2');
```

### Set screen resolution with `setScreenResolution()`

If your code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```javascript
t.setScreenResolution( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```javascript
t.setScreenResolution(1366, 768);
```

### Set viewport dimensions with `setViewport()`

If your code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```javascript
t.setViewport( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```javascript
t.setViewport(300, 200);
```

### Set color depth with `setColorDepth()`

If your code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```javascript
t.setColorDepth( {{BITS PER PIXEL}} );
```

The number should be a positive integer, in bits per pixel. Example:

```javascript
t.setColorDepth(32);
```

### Set the timezone with `setTimezone()`

This method lets you pass a user's language in to Snowplow:

```javascript
t.setTimezone( {{TIMEZONE}} );
```

Example:

```javascript
t.setTimezone('Europe/London');
```

### Set the language with `setLang()`

This method lets you pass a user's language in to Snowplow:

```javascript
t.setLang( {{LANGUAGE}} );
```

The number should be a positive integer, in bits per pixel. Example:

```javascript
t.setLang('en');
```

### Set the IP Address with `setIpAddress()`

This method lets you set an IP Address on your events:

```javascript
t.setIpAddress( {{IP ADDRESS}} );
```

Example:

```javascript
t.setIpAddress('192.168.0.1');
```

### Set the Useragent with `setUseragent()`

This method lets you override the default Useragent on your events:

```javascript
t.setUseragent( {{USER AGENT}} );
```

Example:

```javascript
t.setUseragent('myapp/0.1.0');
```

---

# Node.js tracker SDK
> Track events in Node.js applications with the Snowplow Node.js tracker SDK written in TypeScript.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/

The Snowplow Node.js Tracker allows you to track Snowplow events from your Node.js applications.

The tracker should be straightforward to use if you are comfortable with JavaScript development; any prior experience with other Snowplow trackers is helpful but not necessary. It is written in TypeScript.

The Node.js tracker, along with the [web trackers](/docs/sources/web-trackers/), are part of [one monorepo](https://github.com/snowplow/snowplow-javascript-tracker). The current version is 4.8.0.

---

# Install and initialize the Node.js tracker
> Install and initialize the Node.js tracker using npm with emitter configuration for sending events to Snowplow collectors.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/initialization/

The Snowplow Node.js Tracker is tested and compatible with Node.js versions from 18 to 20. Installation requires [npm](https://www.npmjs.org/), [pnpm](https://pnpm.js.org/) or [yarn](https://yarnpkg.com/).

## Installation

Setting up the tracker should be straightforward if you are familiar with npm:

```bash
npm install @snowplow/node-tracker
```

alternative you can use pnpm or yarn:

```bash
pnpm add @snowplow/node-tracker
```

```bash
yarn add @snowplow/node-tracker
```

The Snowplow Node.js Tracker is also **bundled with TypeScript types and interfaces** so will integrate easily with TypeScript applications.

## Initialization

Require the Node.js Tracker module into your code like so:

```javascript
const snowplow = require('@snowplow/node-tracker');

const tracker = snowplow.newTracker( /* ... */ );
```

or, if using ES Modules, you can import the module like so:

```javascript
import { newTracker } from '@snowplow/node-tracker';
```

The `newTracker` call takes 2 arguments – the tracker and emitter configuration. The tracker configuration specifies the tracker namespace and app name. The emitter configuration specifies how events are queued locally and sent to the Snowplow collector.

```javascript
const tracker = newTracker({

  namespace: "my-tracker", // Namespace to identify the tracker instance. It will be attached to all events which the tracker fires.

  appId: "my-app", // Application identifier

  encodeBase64: false, // Whether to use base64 encoding for the self-describing JSON. Defaults to true.

}, {

  endpoint: "https://collector.mydomain.net", // Collector endpoint

  eventMethod: "post", // Method - defaults to POST

  bufferSize: 1, // Only send events once n are buffered. Defaults to 1 for GET requests and 10 for POST requests.

});
```

There are a number of additional parameters that may optionally be configured. Please refer to the [API docs](https://github.com/snowplow/snowplow-javascript-tracker/blob/master/api-docs/docs/node-tracker/node-tracker.api.md) for the full list.

### Using multiple Emitters

In case you want to send the events to multiple Snowplow collectors, you can provide a list of emitter configurations in the `newTracker` call – one for each collector. This may look as follows.

```javascript
const tracker = newTracker({

  namespace: "my-tracker", // Namespace to identify the tracker instance. It will be attached to all events which the tracker fires.

  appId: "my-app", // Application identifier

  encodeBase64: false, // Whether to use base64 encoding for the self-describing JSON. Defaults to true.

}, [

  {

    endpoint: "https://collector1.mydomain.net", // First collector endpoint

  },

  {

    endpoint: "https://collector2.mydomain.net", // Second collector endpoint

  },

]);
```

### Create your own Emitter

Emitter is an object responsible for queuing tracked events and making requests to the Snowplow collector.

By default, the Node tracker makes use of the [fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) to make the HTTP requests. If you want to use a different library or logic for queuing and sending events, you can provide a custom `Emitter` implementation.

Emitters must conform to an [`Emitter` interface](https://github.com/snowplow/snowplow-javascript-tracker/blob/master/trackers/node-tracker/docs/markdown/node-tracker.emitter.md), which looks like:

```typescript
/**

 * Emitter is responsible for sending events to the collector.

 * It manages the event queue and sends events in batches depending on configuration.

 */

interface Emitter {

    /**

     * Forces the emitter to send all events in the event store to the collector.

     * @returns A Promise that resolves when all events have been sent to the collector.

     */

    flush: () => Promise<void>;

    /**

     * Adds a payload to the event store or sends it to the collector.

     * @param payload - A payload to be sent to the collector

     * @returns Promise that resolves when the payload has been added to the event store or sent to the collector

     */

    input: (payload: Payload) => Promise<void>;

    /**

     * Updates the collector URL to which events will be sent.

     * @param url - New collector URL

     */

    setCollectorUrl: (url: string) => void;

    /**

     * Sets the server anonymization flag.

     */

    setAnonymousTracking: (anonymous: boolean) => void;

    /**

     * Updates the buffer size of the emitter.

     */

    setBufferSize: (bufferSize: number) => void;

}
```

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

Once you implement your own `Emitter`, you can pass it to the `newTracker` call as follows.

```ts
export const expressTracker = newTracker(

  {

    namespace: "my-tracker",

    appId: "my-app",

  },

  {

    customEmitter: () => ({

      input: (payload) => {

        return Promise.resolve();

      },

      flush: () => {

        return Promise.resolve();

      },

      setCollectorUrl: (url) => {},

      setAnonymousTracking: (anonymous) => {},

      setBufferSize: (bufferSize) => {},

    }),

  }

);
```

---

# Migration guides for the Node.js tracker
> Upgrade guides to help you migrate between major versions of the Node.js tracker.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/migration-guides/

This section contains information to help you upgrade to newer versions of the Node.js tracker.

---

# Migration guide for Node.js tracker v3 to v4
> Migration guide for upgrading the Node.js tracker from version 3 to version 4 with breaking changes and new features.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/migration-guides/v3-to-v4-migration-guide/

Version 4 is a major release with several breaking changes to defaults and behavior.

## Behavior changes

### Removed the use of the got library in favor of fetch

The tracker no longer needs the got library for making HTTP requests to the collector. This has been replaced with the [fetch API](https://nodejs.org/dist/latest-v18.x/docs/api/globals.html). The change has an effect on the public API of the tracker which had to previously accommodate the got library – the following change results from this.

### Removed `tracker` and `emitter` functions in favor or `newTracker`

The initialization of a tracker instance has changed – instead of initializing an emitter (using `gotEmitter`) and tracker (using `tracker`) separately, the SDK handles initialization of both using the `newTracker` call. The `newTracker` call now accepts configuration objects for the tracker and emitter as arguments.

[See the new initialization options here.](/docs/sources/node-js-tracker/initialization/)

### Dropped support for older Node.JS

The support for the following Node.JS versions has been dropped:

- Drop Node.js <18 support

---

# Track events with the Node.js tracker
> Track page views, screen views, structured events, and custom self-describing events with the Node.js tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/tracking-events/

Tracking methods supported by the Node.js tracker at a glance:

| **Function**                                                                               | **Description**                                        |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| [`track()`](#track-events-with-the-track-function)                                         | Tracks an event generated from a build functions       |
| [`buildScreenView()`](#track-screen-views-withbuildscreenview)                             | Track the user viewing a screen within the application |
| [`buildPageView()`](#track-pageviews-withbuildpageview)                                    | Track and record views of web pages.                   |
| [`buildSelfDescribingEvent()`](#track-self-describing-events-withbuildselfdescribingevent) | Track a Snowplow custom self describing event          |
| [`buildStructEvent()`](#track-structured-events-with-buildstructevent)                     | Track a Snowplow custom structured event               |

### Track events with the track() function

All events are tracked through the `track()` function on the tracker instance. The type of event is determined by one of the event builder functions, which all begin as `build*()`.

#### Optional arguments

Each build method which is passed to the `track()` function has both default and optional arguments. If you don't want to provide a value for an optional argument, just ignore it in the object or pass `null` and it will be ignored. For example, if you want to track a page view event with a referrer but without a title:

```javascript
t.track(buildPageView({

  pageUrl: 'http://www.example.com',

  referrer: 'http://www.referer.com'

});
```

#### Custom contexts

In short, custom contexts let you add additional information about the circumstances surrounding an event. Each call to `track()` accepts an additional optional context parameter:

```javascript
t.track(buildPageView({

  pageUrl: 'http://www.example.com',

  referrer: 'http://www.referer.com'

}),

[]); // Array of custom contexts
```

The `context` argument should consist of an array of zero or more context objects. Each object should be a self-describing JSON following the same pattern as an self describing event.

**Important:** Even if only one custom context is being attached to an event, it still needs to be wrapped in an array.

If a visitor arrives on a page advertising a movie, the context dictionary might look like this:

```json
{

  "schema": "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

  "data": {

    "movie_name": "Solaris",

    "poster_country": "JP"

  }

}
```

This is how to fire a page view event with the above custom context:

```javascript
t.track(buildPageView({

  pageUrl: 'http://www.example.com',

  referrer: 'http://www.referer.com'

}),

[{

  "schema": "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

  "data": {

    "movie_name": "Solaris",

    "poster_country": "JP"

  }

}]);
```

Note that even though there is only one custom context attached to the event, it still needs to be placed in an array.

#### Timestamps

Each `track()` call supports an optional timestamp as its final argument, after the context argument; this allows you to manually override the timestamp attached to this event.

If you do not pass this timestamp in as an argument, then the Tracker will use the current time to be the timestamp for the event.

Here is an example tracking a structured event and supplying the optional timestamp argument:

```javascript
t.track(buildStructEvent({

  category: "game action",

  action: "save"

}), [], 1368725287000);
```

Timestamp is counted in milliseconds since the Unix epoch - the same format as generated by `new Date().getTime()`.

### Track screen views with `buildScreenView()`

Use `buildScreenView()` to build a user viewing a screen (or equivalent) within your app. Arguments are:

| **Argument** | **Description**                     | **Required?** | **Type**         |
| ------------ | ----------------------------------- | ------------- | ---------------- |
| `name`       | Human-readable name for this screen | No            | Non-empty string |
| `id`         | Unique identifier for this screen   | No            | String           |

`name` and `id` are not individually required, but you must provide at least one of them.

Example:

```javascript
t.track(buildScreenView({

  name: "HUD > Save Game",

  id: "screen23"

}));
```

### Track pageviews with `buildPageView()`

Use `buildPageView()` to track a user viewing a page within your app. Arguments are:

| **Argument** | **Description**                      | **Required?** | **Type**         |
| ------------ | ------------------------------------ | ------------- | ---------------- |
| `pageUrl`    | The URL of the page                  | Yes           | Non-empty string |
| `pageTitle`  | The title of the page                | No            | String           |
| `referrer`   | The address which linked to the page | No            | String           |

Example:

```javascript
t.track(buildPageView({

  pageUrl: "www.example.com",

  url: "example",

  referrer: "www.referrer.com"

}));
```

### Track self describing events with `buildSelfDescribingEvent()`

Use `buildSelfDescribingEvent()` to track a custom event which consists of a name and an unstructured set of properties. This is useful when you want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow).

The arguments are as follows:

| **Argument** | **Description**                      | **Required?** | **Type** |
| ------------ | ------------------------------------ | ------------- | -------- |
| `event`      | The self describing event definition | Yes           | JSON     |

Example:

```javascript
t.track(buildSelfDescribingEvent({

  event: {

    schema: "iglu:com.example_company/save-game/jsonschema/1-0-2",

    data: {

      save_id: "4321",

      level: 23,

      difficultyLevel: "HARD",

      dl_content: true

    }

  }

}));
```

The `event` property must be an object with two fields: `schema` and `data`. `data` is an object containing the properties of the self describing event. `schema` identifies the JSON schema against which `data` should be validated.

### Track structured events with `buildStructEvent()`

Use `buildStructEvent()` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument** | **Description**                                                  | **Required?** | **Type**         |
| ------------ | ---------------------------------------------------------------- | ------------- | ---------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes           | Non-empty string |
| `action`     | Defines the type of user interaction which this event involves   | Yes           | Non-empty string |
| `label`      | A string to provide additional dimensions to the event data      | No            | String           |
| `property`   | A string describing the object or the action performed on it     | No            | String           |
| `value`      | A value to provide numerical data about the event                | No            | Number           |

Example:

```javascript
t.track(buildStructEvent({

  category: "shop",

  action: "add-to-basket",

  label: "product",

  property: "pcs",

  value: 2

}));
```

---

# Configure emitters in the PHP tracker
> Configure sync, socket, curl, and file emitters for sending events to Snowplow collectors with GET or POST requests.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/emitters/

We currently support four different emitters: sync, socket, curl, and an out-of-band file emitter. The most basic emitter only requires you to select the type of emitter to be used and specify the Collector's hostname as parameters.

All emitters support both `GET` and `POST` as methods for sending events to Snowplow Collectors.

For the sake of performance, we recommend using `POST` as the tracker can then batch many events together into a single request. Note that depending on your pipeline architecture, your Collector may have limits on the maximum request size it accepts that could be exceeded by large batch sizes.

It is recommended that after you have finished logging all of your events to call the following method:

```php
$tracker->flushEmitters();
```

This empties the event buffers of all emitters associated with your tracker object and sends any left over events. In future releases, this may be an automatic process but for now, it remains manual.

### Sync

The Sync emitter is a basic synchronous emitter which supports both `GET` and `POST` request types.

By default, this emitter uses the Request type POST, HTTP, and a buffer size of 50.

As of version 0.7.0, the emitter has the capability to retry failed requests. In case connection to the Collector can't be established or the request fails with a 4xx (except for 400, 401, 403, 410, 422) or 5xx status code, the same request is retried. The number of times a request should be retried is configurable but defaults to 1. The default back-off period between subsequent retries, starts with 100 ms (configurable) and increases exponentially.

Example emitter creation:

```php
$emitter = new SyncEmitter($collector_uri, "http", "POST", 50);
```

Whilst you can force the buffer size to be greater than 1 for a GET Request; it doesn't yield any performance changes as we can still only send 1 event at a time.

Constructor:

```php
public function __construct($uri, $protocol = NULL, $type = NULL, $buffer_size = NULL, $debug = false, $max_retry_attempts = NULL, $retry_backoff_ms = NULL, $server_anonymization = false)
```

Arguments:

| **Argument**            | **Description**                                                                                                                     | **Required?** | **Validation**   |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | ---------------- |
| `$uri`                  | Collector hostname                                                                                                                  | Yes           | Non-empty string |
| `$protocol`             | Collector Protocol (HTTP or HTTPS)                                                                                                  | No            | String           |
| `$type`                 | Request Type (POST or GET)                                                                                                          | No            | String           |
| `$buffer_size`          | Amount of events to store before flush                                                                                              | No            | Int              |
| `$debug`                | Whether or not to log errors                                                                                                        | No            | Boolean          |
| `$max_retry_attempts`   | The maximum number of times to retry a request. Defaults to 1.                                                                      | No            | Int              |
| `$retry_backoff_ms`     | The number of milliseconds to backoff before retrying a request. Defaults to 100 ms, increases exponentially in subsequent retries. | No            | Int              |
| `$server_anonymization` | Enable Server Anonymization for sent events; IP and Network User ID information isn't associated with tracked events                | No            | Int              |

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

### Socket

The Socket emitter allows for the much faster transmission of Requests to the Collector by allowing us to write data directly to the HTTP socket. However, this solution is still, in essence, a synchronous process and blocks the execution of the main script.

As of version 0.7.0, the emitter has the capability to retry failed requests. In case connection to the Collector can't be established or the request fails with a 4xx (except for 400, 401, 403, 410, 422) or 5xx status code, the same request is retried. The number of times a request should be retried is configurable but defaults to 1. The default back-off period between subsequent retries, starts with 100 ms (configurable) and increases exponentially.

Example Emitter creation:

```php
$emitter = new SocketEmitter($collector_uri, NULL, "GET", NULL, NULL);
```

Whilst you can force the buffer size to be greater than 1 for a GET Request; it does not yield any performance changes as we can still only send 1 event at a time.

Constructor:

```php
public function __construct($uri, $ssl = NULL, $type = NULL, $timeout = NULL, $buffer_size = NULL, $debug = NULL, $max_retry_attempts = NULL, $retry_backoff_ms = NULL)
```

Arguments:

| **Argument**            | **Description**                                                                                                                     | **Required?** | **Validation**   |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | ---------------- |
| `$uri`                  | Collector hostname                                                                                                                  | Yes           | Non-empty string |
| `$ssl`                  | Whether to use SSL encryption                                                                                                       | No            | Boolean          |
| `$type`                 | Request Type (POST or GET)                                                                                                          | No            | String           |
| `$timeout`              | Socket Timeout Limit                                                                                                                | No            | Int or Float     |
| `$buffer_size`          | Amount of events to store before flush                                                                                              | No            | Int              |
| `$debug`                | Whether or not to log errors                                                                                                        | No            | Boolean          |
| `$max_retry_attempts`   | The maximum number of times to retry a request. Defaults to 1.                                                                      | No            | Int              |
| `$retry_backoff_ms`     | The number of milliseconds to backoff before retrying a request. Defaults to 100 ms, increases exponentially in subsequent retries. | No            | Int              |
| `$server_anonymization` | Enable Server Anonymization for sent events; IP and Network User ID information isn't associated with tracked events                | No            | Int              |

### Curl

The Curl Emitter allows us to have the closest thing to native asynchronous requests in PHP. The curl emitter uses the `curl_multi_init` resource which allows us to send any number of requests asynchronously. This garners a performance gain over the sync and socket emitters as we can now send more than one request at a time.

On top of this, we are also using a modified version of this **[Rolling Curl library](https://github.com/joshfraser/rolling-curl)** for the actual sending of the curl requests. This allows for a more efficient implementation of asynchronous curl requests as we can now have multiple requests sending at the same time, and in addition as soon as one is done a new request is started.

> **Note:** The emitter doesn't retry failed requests to the Collector. Failed requests to the Collector (for example due to it being not reachable) result in lost events.

Example Emitter creation:

```php
$emitter = new CurlEmitter($collector_uri, false, "GET", 2);
```

Whilst you can force the buffer size to be greater than 1 for a GET request, it doesn't yield any performance changes as we can still only send 1 event at a time.

Constructor:

```php
public function __construct($uri, $protocol = NULL, $type = NULL, $buffer_size = NULL, $debug = false, $curl_timeout = NULL)
```

Arguments:

| **Argument**            | **Description**                                         | **Required?** | **Validation**   |
| ----------------------- | ------------------------------------------------------- | ------------- | ---------------- |
| `$uri`                  | Collector hostname                                      | Yes           | Non-empty string |
| `$protocol`             | Collector Protocol (HTTP or HTTPS)                      | No            | String           |
| `$type`                 | Request Type (POST or GET)                              | No            | String           |
| `$buffer_size`          | Amount of events to store before flush                  | No            | Int              |
| `$debug`                | Whether or not to log errors                            | No            | Boolean          |
| `$curl_timeout`         | Maximum time the request is allowed to take, in seconds | No            | Int              |
| `$server_anonymization` | Enable Server Anonymization for sent events             | No            | Int              |

#### Curl default settings

The internal emitter default settings are as follows:

- Rolling Window (max number of concurrent requests)

  - POST: 10
  - GET: 30

- Curl Buffer (number of times we need to hit the emitters buffer size before sending)

  - POST: 50
  - GET: 250

Since version 0.8 of the PHP tracker, you can change these settings using the following setter functions:

- `$curl_emitter.setCurlAmount($curl_amount)`: update the curl buffer size (number of times we need to reach the buffer size before we initiate sending)
- `$curl_emitter.setRollingWindow($rolling_window)`: update the rolling window configuration (max number of concurrent requests)

### File

> **Warning:** When running under Windows, PHP can't spawn truly separate processes, and slowly eats more and more resources when more processes are spawned. Thus, Windows might crash under high load when using the File Emitter.

The File Emitter is the only true non-blocking solution. The File Emitter works via spawning workers which grab created files of logged events from a local temporary folder. The workers then load the events using the same asynchronous curl properties from the above emitter.

All of the worker processes are created as background processes so none of them delay the execution of the main script. Currently, they're configured to look for files inside created worker folders until there are none left and they hit their `timeout` limit, at which point the process terminates itself.

If the worker for any reason fails to successfully send a request it renames the entire file to `failed` and leaves it in the `/temp/failed-logs/` folder.

> **Note:** The emitter doesn't retry failed requests to the Collector. Failed requests to the Collector (for example due to it being not reachable) result in lost events.

Example Emitter creation:

```php
$emitter = new FileEmitter($collector_uri, false, "POST", 2, 15, 100, "/tmp/snowplow");
```

The buffer for the file emitter works a bit differently to the other emitters in that here it refers to the number of events needed before an `events-random.log` is produced for a worker. If you are anticipating it taking a long time to reach the buffer be aware that the worker terminates itself after 75 seconds by default (15 x 5). Adjust the timeout amount in the construction of the FileEmitter if the default isn't suitable.

Constructor:

```php
public function __construct($uri, $protocol = NULL, $type = NULL, $workers = NULL, $timeout = NULL, $buffer_size = NULL, $debug = false, $log_dir = NULL)
```

Arguments:

| **Argument**   | **Description**                                                            | **Required?** | **Validation**   |
| -------------- | -------------------------------------------------------------------------- | ------------- | ---------------- |
| `$uri`         | Collector hostname                                                         | Yes           | Non-empty string |
| `$protocol`    | Collector Protocol (HTTP or HTTPS)                                         | No            | String           |
| `$type`        | Request Type (POST or GET)                                                 | No            | String           |
| `$workers`     | Amount of background workers                                               | No            | Int              |
| `$timeout`     | Worker Timeout                                                             | No            | Int or Float     |
| `$buffer_size` | Amount of events to store before flush                                     | No            | Int              |
| `$debug`       | Whether or not to log errors                                               | No            | Boolean          |
| `$log_dir`     | The directory for event log and worker log subdirectories to be created in | No            | String           |

### Emitter debug mode

Debug mode is enabled on emitters by setting the `$debug` argument in the emitter constructor to `true`:

```php
$emitter = new SyncEmitter($collector_uri, "http", "POST", 50, true);
```

By default, debug mode creates a new directory called `/debug/` in the root of the tracker's directory. It then creates a log file with the following structure; `sync-events-log-[[random number]].log`: which is the type of emitter and a randomized number to prevent it from being accidentally overwritten.

If physically storing the information isn't possible due to not having the correct write permissions or simply not wanted it can be turned off by updating the following value in the Constants class:

```php
const DEBUG_LOG_FILES = false;
```

Now all debugging information is printed to the console.

Every time the events buffer is flushed we can see if the flush was successful. In the case of an error, it records the entire event payload the tracker was trying to send, along with the error code.

#### Event specific information

If debug mode is enabled the emitter begins storing information internally. It stores the HTTP response code and the payload for every request made by the emitter.

```php
array(

    "code" => 200,

    "data" => "{"e":"pv","url":"www.example.com","page":"example","refr":"www.referrer.com"}"

)
```

The `data` is stored as a JSON-encoded string. To locally test whether or not your emitters are successfully sending, we can retrieve this information with the following commands:

```php
$emitters = $tracker->returnEmitters(); # Store all of the emitters as an array.

$emitter = $emitters[0]; # Get the first emitter stored by the tracker

$results = $emitter->returnRequestResults();  # Return the stored results.



# Now that we have results we can work with...

print("Code: ".$results[0]["code"]);

print("Data: ".$results[0]["data"]);
```

This allows you to debug on a request by request basis to ensure that everything is being sent properly.

#### Turn debug off

As debugging stores a lot of information, we can end debug mode by calling the following command:

```php
$tracker->turnOffDebug();
```

This stops all logging activity, both to the external files and to the local arrays. We can go one step further though and pass a `true` boolean to the function. This deletes all of the tracker's associated physical debug log files as well as emptying the local arrays within each linked emitter.

```php
$tracker->turnOffDebug(true);
```

---

# PHP tracker SDK
> Track events in PHP applications and scripts with the Snowplow PHP tracker SDK using trackers, subjects, and emitters.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/

The Snowplow PHP Tracker allows you to track Snowplow events from your PHP apps and scripts.

There are three basic types of object you will create when using the Snowplow PHP Tracker: Trackers, Subjects and Emitters.

A subject represents a user whose events are tracked. A tracker constructs events and sends them to one or more emitters. Each emitter then sends the event to the endpoint you configure, a Snowplow collector.

The current flow of the PHP Tracker is illustrated below:

![Flowchart showing the Snowplow PHP Tracker architecture: the Tracker contains one or more Emitters and one Subject; Emitters (SyncEmitter, CurlEmitter, SocketEmitter) buffer events and send them to the Collector; FileEmitter spawns Worker processes that consume event log files and send them to the Collector, with failed event logs copied to a failed folder](/assets/images/php-tracker-flow-0a55defa12a400416329dcf1e75a4143.png)

---

# Initialize the PHP tracker
> Initialize the PHP tracker with emitter, subject, namespace, and encoding configuration for event tracking.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/initialization/

To instantiate a new Tracker instance we need to make sure the Snowplow Tracker classes are available.

Include these class aliases in your project:

```php
use Snowplow\Tracker\Tracker;

use Snowplow\Tracker\Subject;

use Snowplow\Tracker\Emitters\SyncEmitter;
```

We can now create our Emitter, Subject and Tracker objects.

### Creating a Tracker

The most basic Tracker instance will only require you to provide an [Emitter](/docs/sources/php-tracker/emitters/) and the data [Subject](/docs/sources/php-tracker/subjects/) for which the Tracker will log events.

```php
$emitter = new SyncEmitter($collector_uri, "http", "POST", 10, false);

$subject = new Subject();

$tracker = new Tracker($emitter, $subject);
```

Other Tracker arguments:

| **Argument Name** | **Description**                                                            | **Required?** | **Default** |
| ----------------- | -------------------------------------------------------------------------- | ------------- | ----------- |
| `emitters`        | The emitter to which events are sent                                       | Yes           | `None`      |
| `subject`         | The user being tracked                                                     | Yes           | `Subject()` |
| `namespace`       | The name of the tracker instance                                           | No            | `None`      |
| `app_id`          | The application ID                                                         | No            | `None`      |
| `encode_base64`   | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64) | No            | `True`      |

Another example using all of the allowed arguments:

```php
$tracker = new Tracker($emitter, $subject, "cf", "cf29ea", true);
```

#### `emitters`

This can be either a single emitter or an array of emitters. The tracker will send events to all of these emitters, which will, in turn, send them on to a collector.

```php
$emitter1 = new SyncEmitter($collector_uri);

$emitter2 = new CurlEmitter($collector_uri, false, "GET", 2);



$emitter_array = array($emitter1, $emitter2);



// Tracker Init

$subject = new Subject();

$tracker1 = new Tracker($emitter1, $subject); # Single Emitter

$tracker2 = new Tracker($emitter_array, $subject); # Array of Emitters
```

#### `subject`

The user which the Tracker will track. This will give your events user-specific data such as timezone and language. You change the subject of your tracker at any time by calling `updateSubject($new_subject_object)`. All events sent from this Tracker will now have the new subject information attached.

#### `namespace`

If provided, the `namespace` argument will be attached to every event fired by the new tracker. This allows you to later identify which tracker fired which event if you have multiple trackers running.

#### `app_id`

The `app_id` argument lets you set the application ID to any string.

#### `encode_base64`

By default, unstructured events and custom contexts are encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the `encode_base64` argument.

---

# Install the PHP tracker
> Install the PHP tracker using Composer for tracking events in PHP applications.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/setup/

Currently the only supported method of installation is through Composer. For a tutorial on setting up a PHP project with Composer please follow this [link](https://getcomposer.org/doc/00-intro.md).

Add the Snowplow PHP Tracker to your project by running:

```bash
composer require snowplow/snowplow-tracker
```

to include it in your `composer.json` file as a dependency.

You can also add it manually:

```json
{

  "require": {

      "snowplow/snowplow-tracker": "0.9.2"

  }

}
```

Assuming you have Composer setup correctly in the root of your project. Type the following command line argument:

```bash
composer update # Will update lockfile and install dependencies
```

This will install the Snowplow Tracker and allow you to initialize a Tracker object:

```php
// Bare minimum Tracker initialization.



use Snowplow\Tracker\Tracker;

use Snowplow\Tracker\Subject;

use Snowplow\Tracker\Emitters\SyncEmitter;



$emitter = new SyncEmitter("collector_uri");

$subject = new Subject();

$tracker = new Tracker($emitter, $subject);
```

---

# Track subject data with the PHP tracker
> Add user and environment data to tracked events using the Subject class with platform, user ID, and device properties.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/subjects/

The Subject object lets you send any additional information about your application's environment, current user, etc to Snowplow.

To create a new subject:

```php
$subject = new Subject();
```

By default the subject has one piece of information in it already, the [platform](/docs/fundamentals/canonical-event/#application-fields) `["p" => "srv"]`.

The Subject class contains a variety of 'set' methods to attach extra data to your event.

- [`setPlatform`](#setplatform)
- [`setUserId`](#setuserid)
- [`setScreenResolution`](#setscreenresolution)
- [`setViewPort`](#setviewport)
- [`setColorDepth`](#setcolordepth)
- [`setTimezone`](#settimezone)
- [`setLanguage`](#setlanguage)
- [`setIpAddress`](#setipaddress)
- [`setUseragent`](#setuseragent)
- [`setNetworkUserId`](#setnetworkuserid)
- [`setDomainUserId`](#setdomainuserid)
- [`setSessionId`](#setsessionid)
- [`setSessionIndex`](#setsessionindex)
- [`setRefr`](#setrefr)
- [`setPageUrl`](#setpageurl)

These set methods can be called either directly onto a subject object:

```php
$subject = new Subject();

$subject->setPlatform("tv");
```

Or they can be called through the tracker object:

```php
$tracker->returnSubject()->setPlatform("tv");
```

You can also change the active Subject a Tracker is operating with:

```php
$subject2 = new Subject();

$tracker->updateSubject($subject2);
```

### `setPlatform`

The default platform is "srv". You can change the platform of the subject by calling:

```php
$subject->setPlatform($platform);
```

For example:

```php
$subject->setPlatform("tv") # Running on a Connected TV
```

For a full list of supported platforms, please see the [Event Parameters reference](/docs/fundamentals/canonical-event/#application-fields).

### `setUserId`

You can set the user ID to any string:

```php
$subject->setUserId($id);
```

Example:

```php
$subject->setUserId("jbeem");
```

### `setScreenResolution`

If your PHP code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```php
$subject->setScreenResolution($width, $height);
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```php
$subject->setScreenResolution(1366, 768);
```

### `setViewPort`

If your PHP code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```php
$subject->setViewPort($width, $height);
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```php
$subject->setViewPort(300, 200);
```

### `setColorDepth`

If your PHP code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```php
$subject->setColorDepth($depth);
```

The number should be a positive integer, in bits per pixel. Example:

```php
$subject->setColorDepth(32);
```

### `setTimezone`

This method lets you pass a user's timezone in to Snowplow:

```php
$subject->setTimezone($timezone);
```

The timezone should be a string:

```php
$subject->setTimezone("Europe/London");
```

### `setLanguage`

This method lets you pass a user's language in to Snowplow:

```php
$subject->setLanguage($language);
```

The language should be a string:

```php
$subject->setLanguage('en');
```

### `setIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```php
$subject->setIpAddress($ip);
```

The IP Address should be a string:

```php
$subject->setIpAddress('127.0.0.1');
```

### `setUseragent`

This method lets you pass a useragent in to Snowplow:

```php
$subject->setUseragent($useragent);
```

The useragent should be a string:

```php
$subject->setUseragent('Agent Smith');
```

### `setNetworkUserId`

This method lets you pass a Network User ID in to Snowplow:

```php
$subject->setNetworkUserId($networkUserId);
```

The network user id should be a string:

```php
$subject->setNetworkUserId("network-id");
```

### `setDomainUserId`

This method lets you pass a Domain User ID in to Snowplow:

```php
$subject->setDomainUserId($domainUserId);
```

The domain user id should be a string:

```php
$subject->setDomainUserId("domain-id");
```

### `setSessionId`

This method lets you pass a session ID in to Snowplow:

```php
$subject->setSessionId($sessionId);
```

The session ID should be a string:

```php
$subject->setSessionId("759e1c9a-6b74-403c-8b6f-18eb9f0c2f02");
```

### `setSessionIndex`

This method lets you pass a session index in to Snowplow:

```php
$subject->setSessionIndex($sessionIndex);
```

The session index should be a number:

```php
$subject->setSessionIndex(1);
```

### `setRefr`

This method lets you set the referrer URL to a full URI:

```php
$subject->setRefr("https://example.com/previous-page");
```

### `setPageUrl`

This method lets you set the page URL to a full URI string:

```php
$subject->setPageUrl("https://example.com/current-page");
```

---

# Track an event with the PHP tracker
> Track page views, ecommerce transactions, screen views, and custom events with the PHP tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/tracking-an-event/

Tracking methods supported by the PHP Tracker:

| **Function**                                              | **Description**                                        |
| --------------------------------------------------------- | ------------------------------------------------------ |
| [`trackPageView`](#trackpageview)                         | Track and record views of web pages.                   |
| [`trackEcommerceTransaction`](#trackecommercetransaction) | Track an ecommerce transaction                         |
| [`trackScreenView`](#trackscreenview)                     | Track the user viewing a screen within the application |
| [`trackStructEvent`](#trackstructevent)                   | Track a Snowplow custom structured event               |
| [`trackUnstructEvent`](#trackunstructevent)               | Track a Snowplow custom unstructured event             |

### Optional Tracking Arguments

#### Custom Context

Custom contexts let you add additional information about any circumstances surrounding an event in the form of a PHP Array of name-value pairs. Each tracking method accepts an additional optional contexts parameter after all the parameters specific to that method:

```php
public function trackPageView($page_url, $page_title = NULL, $referrer = NULL, $context = NULL, $tstamp = NULL)
```

An example of a Context Array Structure:

```php
array(

    "schema" => "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

    "data" => array(

        "movie_name" => "Solaris",

        "poster_country" => "JP"

    )

)
```

This is how to fire a page view event with the above custom context:

```php
$tracker->trackPageView(

    "http://www.films.com",

    "Homepage",

    NULL,

    array(

        array(

            "schema" => "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

            "data" => array(

                "movie_name" => "Solaris",

                "poster_country" => "JP"

            )

        )

    )

);
```

#### Timestamp

Each tracking method supports an optional timestamp as its final argument; this allows you to also set the true timestamp (`true_tstamp`) of the event besides the device created timestamp (`dvce_created_tstamp`) that always gets set by the tracker. The optional timestamp argument should be provided in **milliseconds** since the Unix epoch.

Here is an example tracking a structured event and supplying the optional timestamp argument. We can explicitly supply a `NULL` for the intervening arguments which are empty:

```php
$tracker->trackStructEvent("some cat", "save action", NULL, NULL, NULL, 1368725287000);
```

### Event Tracking Methods

#### `trackPageView`

Track a user viewing a page within your app.

Function:

```php
public function trackPageView($page_url, $page_title = NULL, $referrer = NULL, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument**  | **Description**                      | **Required?** | **Validation**   |
| ------------- | ------------------------------------ | ------------- | ---------------- |
| `$page_url`   | The URL of the page                  | Yes           | Non-empty string |
| `$page_title` | The title of the page                | No            | String           |
| `$referrer`   | The address which linked to the page | No            | String           |
| `$context`    | Custom context for the event         | No            | Array            |
| `$tstamp`     | When the pageview occurred           | No            | Positive integer |

Example Usage:

```php
$tracker->trackPageView("www.example.com", NULL, NULL, NULL, 123123132132);
```

#### `trackEcommerceTransaction`

Track an ecommerce transaction.

Function:

```php
public function trackEcommerceTransaction($order_id, $total_value, $currency = NULL, $affiliation = NULL, $tax_value = NULL, $shipping = NULL, $city = NULL, $state = NULL, $country = NULL, $items = array(), $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument**   | **Description**                     | **Required?** | **Validation**   |
| -------------- | ----------------------------------- | ------------- | ---------------- |
| `$order_id`    | ID of the eCommerce transaction     | Yes           | Non-empty string |
| `$total_value` | Total transaction value             | Yes           | Int or Float     |
| `$currency`    | Transaction currency                | No            | String           |
| `$affiliation` | Transaction affiliation             | No            | String           |
| `$tax_value`   | Transaction tax value               | No            | Int or Float     |
| `$shipping`    | Delivery cost charged               | No            | Int or Float     |
| `$city`        | Delivery address city               | No            | String           |
| `$state`       | Delivery address state              | No            | String           |
| `$country`     | Delivery address country            | No            | String           |
| `$items`       | Items in the transaction            | No            | Array            |
| `$context`     | Custom context for the event        | No            | Array            |
| `$tstamp`      | When the transaction event occurred | No            | Positive integer |

Example Usage:

```php


$tracker->trackEcommerceTransaction(

    "test_order_id_1",

    200,

    "GBP",

    "affiliation_1",

    "tax_value_1",

    "shipping_1",

    "city_1",

    "state_1",

    "country_1",

    array(

        array("name" => "name_1","category" => "category_1",

            "price" => 100,"sku" => "sku_1","quantity" => 1),

        array("name" => "name_2","category" => "category_2",

            "price" => 100,"sku" => "sku_2","quantity" => 1)

    )

);
```

The above example contains an order with two order items.

##### `trackEcommerceTransactionItem`

This is a private function that is called from within `trackEcommerceTransaction`. Note that for an item to be added successfully you need to include the following fields in the array, even if the value is `NULL`.

Arguments:

| **Argument** | **Description** | **Required?** | **Validation**   |
| ------------ | --------------- | ------------- | ---------------- |
| `"sku"`      | Item SKU        | Yes           | Non-empty string |
| `"price"`    | Item price      | Yes           | Int or Float     |
| `"quantity"` | Item quantity   | Yes           | Int              |
| `"name"`     | Item name       | No            | String           |
| `"category"` | Item category   | No            | String           |

Example Item:

```php
array(

    array("name" => NULL,

          "category" => NULL,

          "price" => 100,

          "sku" => "sku_1",

          "quantity" => 1)

)
```

If any of these fields are missing the item event will not be created. However the order of these fields is not important.

#### `trackScreenView`

Track a user viewing a screen (or equivalent) within your app.

Function:

```php
public function trackScreenView($name = NULL, $id = NULL, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument** | **Description**                     | **Required?** | **Validation**   |
| ------------ | ----------------------------------- | ------------- | ---------------- |
| `$name`      | Human-readable name for this screen | No            | Non-empty string |
| `$id`        | Unique identifier for this screen   | No            | String           |
| `$context`   | Custom context for the event        | No            | Array            |
| `$tstamp`    | When the screen was viewed          | No            | Positive integer |

Although `$name` and `$id` are not individually required, at least one must be provided or the event will fail validation.

Example:

```php
$tracker->trackScreenView("HUD > Save Game", NULL, NULL, 1368725287000);
```

#### `trackStructEvent`

Track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required).

Function:

```php
public function trackStructEvent($category, $action, $label = NULL, $property = NULL, $value = NULL, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument** | **Description**                                                  | **Required?** | **Validation**   |
| ------------ | ---------------------------------------------------------------- | ------------- | ---------------- |
| `$category`  | The grouping of structured events which this `action` belongs to | Yes           | Non-empty string |
| `$action`    | Defines the type of user interaction which this event involves   | Yes           | Non-empty string |
| `$label`     | A string to provide additional dimensions to the event data      | No            | String           |
| `$property`  | A string describing the object or the action performed on it     | No            | String           |
| `$value`     | A value to provide numerical data about the event                | No            | Int or Float     |
| `$context`   | Custom context for the event                                     | No            | Array            |
| `$tstamp`    | When the structured event occurred                               | No            | Positive integer |

Example:

```php
$tracker->trackStructEvent("shop", "add-to-basket", NULL, "pcs", 2);
```

#### `trackUnstructEvent`

Track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

Function:

```php
public function trackUnstructEvent($event_json, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument**  | **Description**                      | **Required?** | **Validation**   |
| ------------- | ------------------------------------ | ------------- | ---------------- |
| `$event_json` | The properties of the event          | Yes           | Array            |
| `$context`    | Custom context for the event         | No            | Array            |
| `$tstamp`     | When the unstructured event occurred | No            | Positive integer |

Example:

```php
$tracker->trackUnstructEvent(

    array(

        "schema" => "com.example_company/save-game/jsonschema/1-0-2",

        "data" => array(

            "save_id" => "4321",

            "level" => 23,

            "difficultyLevel" => "HARD",

            "dl_content" => true

        )

    ),

    NULL,

    132184654684

);
```

The `$event_json` must be an array with two fields: `schema` and `data`. `data` is a flat array containing the properties of the unstructured event. `schema` identifies the JSON schema against which `data` should be validated.

### Extra Tracker Functions

#### Tracker `flushEmitters`

The `flushEmitters` function can be called after you have successfully created a Tracker with the following function call:

```php
$tracker->flushEmitters();
```

This will tell the tracker to send any remaining events that are left in the buffer to the collector(s).

---

# Pixel tracker tag
> Track views of HTML-only content like emails with the Snowplow Pixel tracker using simple image tags without JavaScript.
> Source: https://docs.snowplow.io/docs/sources/pixel-tracker/

The Pixel tracker is an HTML-only tracking tag (no JavaScript) to track opens / views of HTML content that does not support JavaScript. Examples of use cases include HTML emails.

In a normal JavaScript tag, the name-value pairs of data that are sent through to the Snowplow Collector via the querystring are calculated on the fly. Examples of data points that are calculated like this include `user_id`, or `page_title`.

In an environment where JavaScript isn't permitted, these values need to be set in advance and hardcoded into the tracking tag. For example, if you want to record a different `page_title` for different pages, you'll need to generate a different tracking tag for each of those pages, with the right `page_title` set for each.

## Example Pixel tracking tags

The structure of a Pixel tracking tag is as follows:

```html
<img src="{{Collector-domain}}/i?{{name-value-pairs}}&tv=no-js-0.1.0" />
```

Some things to note about the tag:

- It's an `<img>` tag
- It sends a GET request using the Snowplow Collector `/i` endpoint
- You can send any name-value pairs that are supported by the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/) on the query string
  - The pipeline will drop name-value pairs that aren't supported by the Snowplow Tracker Protocol
- To help with analysis, we recommend you set the tracker version `tv` to `no-js-0.1.0` to identify that this event came from a Pixel tracker

### Page view tag

Here's an example tag to track a [page view](/docs/events/ootb-data/page-and-screen-view-events/) event:

```html
<!--Snowplow start plowing-->

<img src="http://Collector.acme.com/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=web&tv=no-js-0.1.0" />

<!--Snowplow stop plowing-->
```

The query string contains these key-value pairs:

- Event type `e` is `pv`, for page view event
- Page name `page` is `Root README`
- URL `url` is `http://github.com/snowplow/snowplow`
- Application id `aid` is `snowplow`
- Platform `p` is `web`
- Tracker version `tv` is `no-js-0.1.0`

### Custom self-describing event tag

Here's an example tag to track a [custom self-describing](/docs/events/custom-events/#custom-events) event. This tag tracks an event with the schema `iglu:com.acme/email_open/jsonschema/1-0-0` and the data payload:

```json
{

  "emailId": "abc123",

  "campaign": "spring-sale"

}
```

```html
<!--Snowplow start plowing-->

<img src="http://Collector.acme.com/i?e=ue&ue_pr=%7B%22schema%22%3A%22iglu%3Acom.snowplowanalytics.snowplow%2Funstruct_event%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22schema%22%3A%22iglu%3Acom.acme%2Femail_open%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22emailId%22%3A%22abc123%22%2C%22campaign%22%3A%22spring-sale%22%7D%7D%7D&aid=acme&p=web&tv=no-js-0.1.0" />

<!--Snowplow stop plowing-->
```

The query string contains these key-value pairs:

- Event type `e` is `ue`, for self-describing event (they used to be called "unstructured events")
- Event payload `ue_pr` is the [JSON-encoded self-describing event](/docs/fundamentals/schemas/#self-describing-json-data-in-the-event-payload)
- Application id `aid` is `acme`
- Platform `p` is `web`
- Tracker version `tv` is `no-js-0.1.0`

The full [`ue_pr` value](/docs/events/http-requests/) is structured like this:

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",

  "data": {

    "schema": "iglu:com.acme/email_open/jsonschema/1-0-0",

    "data": {

      "emailId": "abc123",

      "campaign": "spring-sale"

    }

  }

}
```

### Entities

You can also include [entities](/docs/fundamentals/entities/) in your Pixel tracking tags. Here's a tag that extends the page view example to add the `email_open` custom data as an entity:

```html
<!--Snowplow start plowing-->

<img src="http://Collector.acme.com/i?e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&co=%7B%22schema%22%3A%22iglu%3Acom.snowplowanalytics.snowplow%2Fcontexts%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%5B%7B%22schema%22%3A%22iglu%3Acom.acme%2Femail_open%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22emailId%22%3A%22abc123%22%2C%22campaign%22%3A%22spring-sale%22%7D%7D%5D%7D&aid=snowplow&p=web&tv=no-js-0.1.0" />

<!--Snowplow stop plowing-->
```

The entity data is tracked in the [`co` parameter](/docs/fundamentals/schemas/#self-describing-json-data-in-the-event-payload) as a [JSON-encoded array](/docs/events/http-requests/):

```json
{

  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",

  "data": [

    {

      "schema": "iglu:com.acme/email_open/jsonschema/1-0-0",

      "data": {

        "emailId": "abc123",

        "campaign": "spring-sale"

      }

    }

  ]

}
```

## A warning about using the Pixel tracker on domains you don't own

The Snowplow Collector sets a `network_userid` and drops this on a browser cookie.

Care must therefore be exercised when using the Pixel tracker on domains that you do not own. **It is your responsibility to abide by the terms and conditions of any domain owner for domains where you post content including uploading Pixel tracking tags.** Some domain owners forbid third parties from dropping cookies on their domains. It is your responsibility to ensure you do not violate the terms and conditions of any domain owners that you work with.

## Setting up the pixel tracker

### Identify the event you wish to track

Identify the event you wish to track. This may be opening a particular email that is sent out via your email marketing system, or viewing a product you are selling on a third-party marketplace.

### Construct the pixel tracker URL

Build your pixel tracker URL using the format shown above. Set your Collector domain and configure the query string parameters according to the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/).

### Insert the tracking code into the page or ad you wish to track

If this is an HTML email, you will need to insert it in the email. If it is a webpage hosted on a third-party site, you will need to add it to the source code.

## Click tracking

You can use the Pixel tracker for click tracking aka URI redirects:

- Set your Collector path to `{{Collector-domain}}/r/tp2?{{name-value-pairs}}` - the `/r/tp2` tells Snowplow that you are attempting a URI redirect
- Add a `&u={{uri}}` argument to your Collector URI, where `{{uri}}` is the URL-encoded URI that you want to redirect to
- On clicking this link, the Collector will register the link and then do a 302 redirect to the supplied `{{uri}}`
- As well as the `&u={{uri}}` parameter, you can populate the Collector URI with any other fields from the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/)

Redirect tracking is usually disabled by default, and is disabled by default for all Snowplow customers. To use this feature, you need to enable this in your Collector configuration. Snowplow customers can enable this from within the Pipeline Configuration screen of Snowplow Console.

You should also restrict values which are allowed within the `u` parameter to prevent phishing attacks using this redirect endpoint. One option is to use [AWS WAF](https://aws.amazon.com/waf/) or [Google Cloud Armor](https://cloud.google.com/armor) (depending on your cloud). They let you block traffic that matches rules you define, such as a regex that the value of the `u` parameter must match.

Example:

```html
Check out <a href="http://Collector.acme.com/r/tp2?u=https%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow">Snowplow</a>
```

Snowplow converts the `&u={{uri}}` argument into a `com.snowplowanalytics.snowplow/uri_redirect` self-describing JSON.

How Snowplow attaches the `uri_redirect` to the event depends on what other Tracker Protocol fields you attached to the event:

1. If you attached an `&e={{event type}}` to your event, then the `uri_redirect` will be added to the entities array of your event
2. If you did not attach an `&e={{event type}}` to your event, then this event will be treated as an self-describing event and the `uri_redirect` will be attached as the event itself

---

# Type checking in the Python tracker
> Enable runtime validation and type checking for Python tracker arguments using contracts to ensure proper value ranges and types.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/contracts/

Python is a dynamically typed language, but each of our methods expects its arguments to be of specific types and value ranges. To avoid wrong inputs, the tracker takes two approaches:

1. It specifies [type hints](https://docs.python.org/3/library/typing.html) on each function argument and return value. This allows IDEs to validate function calls in your code and also enables static type checking using tools such as [mypy](http://mypy-lang.org/). However, type violations do not result in runtime errors.
2. It validates expected value ranges during runtime and throws `ValueError` errors in case they are violated. Contracts are defined for arguments to check that they belong to expected value ranges such as:

```python
s = Subject()

tracker.set_platform("coffee") # throws ValueError because "coffee" is not one of supported platforms

tracker.set_screen_resolution(width=-1, height=-1) # throws ValueError because width and height arguments have to be greater than 0
```

You can turn off runtime contract validation like this:

```python
from snowplow_tracker import disable_contracts

disable_contracts()
```

---

# Example applications for the Python tracker
> Explore example Python tracker applications including Redis, Snowplow API, and Tracker API implementations for reference and quick start.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/demos/

Check out the following example applications that can be used as a reference for how to use the tracker, or as a starting point for your own implementation.

- [Redis](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/redis_example)
- [Snowplow API](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/snowplow_api_example)
- [Tracker API](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/tracker_api_example)

---

# Configure emitters in the Python tracker
> Configure emitters to send events to Snowplow collectors with batch size, retry logic, and async support using the Emitter and AsyncEmitter classes.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/emitters/

Tracker instances must be initialized with an emitter. This section will go into more depth about the Emitter class and its subclasses.

## The basic Emitter class

At its most basic, the Emitter class only needs a collector URI:

```python
from snowplow_tracker import Emitter



e = Emitter(endpoint="collector.example.com")
```

This is the signature of the constructor for the base Emitter class:

```python
def __init__(

        self,

        endpoint: str,

        protocol: Literal["http", "https"] = "https",

        port: Optional[int] = None,

        method: Literal["get", "post"] = "post",

        batch_size: Optional[int] = None,

        on_success: Optional[Callable[[PayloadDictList], None]] = None,

        on_failure: Optional[Callable[[int, PayloadDictList], None]] = None,

        byte_limit: Optional[int] = None,

        request_timeout: Optional[Union[float, Tuple[float, float]]] = None,

        max_retry_delay_seconds: int = 60,

        buffer_capacity: Optional[int] = None,

        custom_retry_codes: Dict[int, bool] = {},

        event_store: Optional[EventStore] = None,

        session: Optional[requests.Session] = None,

    )-> None:
```

| **Argument**              | **Description**                                                                        | **Required?** | **Type**                                | **Default** |
| ------------------------- | -------------------------------------------------------------------------------------- | ------------- | --------------------------------------- | ----------- |
| `endpoint`                | The collector URI                                                                      | Yes           | String                                  |             |
| `protocol`                | Request protocol: http or https                                                        | No            | String                                  | `https`     |
| `port`                    | The port to connect to                                                                 | No            | Positive integer                        | `None`      |
| `method`                  | The method to use: “get” or “post”                                                     | No            | String                                  | `post`      |
| `batch_size`              | Number of events to store before flushing                                              | No            | Positive integer                        | `10`        |
| `on_success`              | Callback executed when a flush is successful                                           | No            | Function taking 1 argument              | `None`      |
| `on_failure`              | Callback executed when a flush is unsuccessful                                         | No            | Function taking 2 arguments             | `None`      |
| `byte_limit`              | Number of bytes to store before flushing                                               | No            | Positive integer                        | `None`      |
| `request_timeout`         | Timeout for HTTP requests                                                              | No            | Positive integer or tuple of 2 integers | `None`      |
| `max_retry_delay_seconds` | The maximum time between attempts to send failed events to the collector               | No            | int                                     | 60          |
| `buffer_capacity`         | The maximum capacity of the event buffer                                               | No            | int                                     | `None`      |
| `custom_retry_codes`      | Custom retry rules for HTTP status codes received in emit responses from the Collector | No            | dict                                    | `None`      |
| `event_store`             | Stores the event buffer and buffer capacity                                            | No            | EventStore                              | `None`      |
| `session`                 | Persist parameters across requests by using a session object                           | No            | requests.Session                        | `None`      |

> **Note:** If no `event_store` is provided, an `InMemoryEventStore` will be initialized with a `buffer_capacity` of 10,000

See the [`API docs`](https://snowplow.github.io/snowplow-python-tracker/) for more information.

### `protocol`

`protocol` defaults to "https" but also supports "http".

### `batch_size`

When the emitter receives an event, it adds it to a buffer. When the queue is full, all events in the queue get sent to the collector. The `batch_size` argument allows you to customize the queue size. By default, it is 1 for GET requests and 10 for POST requests. (So in the case of GET requests, each event is fired as soon as the emitter receives it.) If the emitter is configured to send POST requests, then instead of sending one for every event in the buffer, it will send a single request containing all those events in JSON format.

### `byte_limit`

`byte_limit` is similar to `batch_size`, but instead of counting events - it takes into account only the amount of bytes to be sent over the network. _Warning_: this limit is approximate with error < 1%.

### `on_success`

`on_success` is an optional callback that will execute whenever the queue is flushed successfully, that is, whenever every request sent has status code 200. It will be passed one argument: an array of events that were successfully sent.

### `on_failure`

`on_failure` is similar, but executes when the flush is not wholly successful. It will be passed two arguments: the number of events that were successfully sent, and an array of unsent events.

An example:

```python
def success(arr):

    print(str(len(arr)) + " events sent successfully!")



def new_success(arr):

    for event_dict in arr:

        print(event_dict)



def failure(num, arr):

    print(str(num) + " events sent successfully!")

    print("These events were not sent successfully:")

    for event_dict in arr:

        print(event_dict)



e = Emitter(endpoint="collector.example.com", batch_size=3, on_success=new_success, on_failure=failure)



t = Tracker(namespace="snowplow_tracker", emitter=e)
```

### `request_timeout`

Timeout for HTTP requests. Can be set either as single float value which applies to both "connect" AND "read" timeout, or as tuple with two float values which specify the "connect" and "read" timeouts separately.

### `max_retry_delay_seconds`

The maximum time between attempts to send failed events to the collector.

### `buffer_capacity`

The maximum capacity of the event buffer. When the buffer is full new events are lost.

### `custom_retry_codes`

Custom retry rules for HTTP status codes received in emit responses from the Collector. By default, retry will not occur for status codes 400, 401, 403, 410 or 422. This can be overridden here by parsing a dictionary of status codes and booleans.

### `event_store`

The event store is used to store an event queue with events scheduled to be sent. Events are added to the event store when they are tracked and removed when they are successfully emitted or when emitting fails without any scheduled retries. The default is an InMemoryEventStore object with a buffer\_capacity of 10,000 events.

### `session`

The session object can be parsed into the emitter to use the requests.Session API. This allows users to persist parameters across requests, as well as pool connections to increase efficiency under heavy usage. If no `session` is parsed, the requests API is used.

## What happens if an event fails to send?

After trying to send a batch of events the collector will return an http status code. A 2xx code is always considered successful. If a failure code is returned (anything other than 2xx, with certain exceptions, see below), the events (as PayloadDictList objects) are returned to the buffer. They will be retried in future sending attempts.

To prevent unnecessary requests being made while the collector is unavailable, an exponential backoff is added to all subsequent event sending attempts. This resets after a request is successful. The default maximum backoff time between attempts is 1 minute but this can be configured by setting `max_retry_delay_seconds`.

The status codes 400 Bad Request, 401 Unauthorised, 403 Forbidden, 410 Gone, or 422 Unprocessable Entity are the exceptions: they are not retried by default. Payloads in requests receiving these responses are not returned to the buffer for retry. They are just deleted.

Configure which codes to retry on or not using the `EmitterConfiguration` when creating your tracker. This method takes a dictionary of status codes and booleans (True for retry and False for not retry).

```python
# by default 401 isn't retried, but 500 is

custom_retry_codes = {500: False, 401: True}

emitter_config = EmitterConfiguration(custom_retry_codes=custom_retry_codes)



Snowplow.create_tracker(

    namespace="ns",

    endpoint="collector.example.com",

    emitter_config=emitter_config,

)
```

## Configuring how events are buffered

As events are collected, the are stored in a buffer until there are enough to send. By default, tracked events are stored in the `InMemoryEventStore`. This is an implemenation of the `EventStore` protocol and stores payloads in a `List` object and is cleared once the buffer capacity is reached.

The default buffer capacity is 10,000 events. This is the number of events that can be stored. When the buffer is full, new tracked payloads are dropped, so choosing the right capacity is important. You can set the buffer capacity through the `EmitterConfiguration` object, for example:

```python
emitter_config = EmitterConfiguration(buffer_capacity=25,000)



Snowplow.create_tracker(

    namespace="ns",

    endpoint="collector.example.com",

    emitter_config=emitter_config,

)
```

The emitter will store 25,000 events before starting to lose data.

## The AsyncEmitter class

```python
from snowplow_tracker import AsyncEmitter



e = AsyncEmitter(endpoint="collector.example.com", thread_count=10)
```

The `AsyncEmitter` class works just like the Emitter class, which is its parent class. It has one advantage, though: HTTP(S) requests are sent asynchronously, so the Tracker won't be blocked while the Emitter waits for a response. For this reason, the AsyncEmitter is recommended over the base `Emitter` class.

The AsyncEmitter uses a fixed-size thread pool to perform network I/O. By default, this pool contains only one thread, but you can configure the number of threads in the constructor using the `thread_count` argument, which is the only specific to AsyncEmitter argument.

Here is a complete example with all constructor parameters set:

```python
from snowplow_tracker import AsyncEmitter



e = AsyncEmitter(

    endpoint="collector.example.com",

    protocol = "https",

    port=9090,

    method='post',

    batch_size=10,

    on_success=None,

    on_failure=None,

    thread_count=1,

    byte_limit=None,

    max_retry_delay_seconds = 60,

    buffer_capacity = 5000,

    event_store = None,

    session=None

)
```

### Manual flushing

You can flush the emitter manually using the `flush` method of the `Tracker` instance which is sending events to the emitter. This is a blocking call which synchronously sends all events in the emitter's buffer.

```python
tracker.flush()
```

You can alternatively perform an asynchronous flush, which tells the tracker to send all buffered events but doesn't wait for the sending to complete:

```python
tracker.flush(False)
```

If you are using the AsyncEmitter, you shouldn't perform a synchronous flush inside an on\_success or on\_failure callback function as this can cause a deadlock.

### Multiple emitters

You can configure a tracker instance to send events to multiple emitters by passing the `Tracker` constructor function an array of emitters instead of a single emitter, or by using the `addEmitter` method:

```python
from snowplow_tracker import Subject, Tracker, AsyncEmitter



e1 = AsyncEmitter(endpoint="collector1.cloudfront.net", method="get")

e2 = AsyncEmitter(endpoint="collector2.cloudfront.net", method="post")



tracker = Tracker(namespace="snowplow_tracker", emitters=[e1, e2])



e3 = AsyncEmitter(endpoint="collector3.cloudfront.net", method="post")



tracker.addEmitter(e3)
```

### Custom emitters

You can create your own custom emitter class, either from scratch or by subclassing one of the existing classes. The only requirement for compatibility is that is must have an `input` method which accepts a Python dictionary of name-value pairs.

### Setting flush timer

You can flush your emitter based on some time interval:

```python
e1 = AsyncEmitter(endpoint="collector1.cloudfront.net", method="post")

e1.set_flush_timer(5)  # flush each 5 seconds
```

Automatic flush can also be cancelled:

```python
e1.cancel_flush()
```

---

# Python tracker SDK
> Track events in Python applications and games with the Snowplow Python tracker SDK using subjects, emitters, and trackers.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/

The Snowplow Python Tracker allows you to track Snowplow events from your Python apps and games.

The tracker should be straightforward to use if you are comfortable with Python development; any prior experience with Snowplow's [JavaScript Tracker](/docs/sources/web-trackers/), Google Analytics or Mixpanel (which have similar APIs to Snowplow) is helpful but not necessary.

There are three basic types of objects you will create when using the Snowplow Python Tracker: subjects, emitters, and trackers.

A subject represents a user whose events are tracked. A tracker constructs events and sends them to one or more emitters. Each emitter then sends the event to the endpoint you configure. This will usually be a Snowplow collector, but could also be a custom event store.

---

# Initialize the Python tracker
> Initialize the Python tracker using the Snowplow class or Tracker, Emitter, and Subject classes with configuration for namespace, endpoint, and encoding.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/intialization/

Assuming you have completed the Python Tracker Setup for your Python project, you are now ready to initialize the Python Tracker. There are two ways to this:

1. Using the `Snowplow` class. This is preferable for most use cases.
2. Using the `Tracker`, `Emitter` and `Subject` classes. This option is useful in case you want to replace one of the internal tracker components (for instance, you want to provide a custom `Emitter` class).

## Option 1: Initialization using the Snowplow class

The Snowplow class contains `static` methods to help manage `Tracker` objects.

Import the Snowplow class along with the required configuration objects:

```python
from snowplow_tracker import Snowplow, EmitterConfiguration, Subject, TrackerConfiguration
```

The simplest tracker configuration can be instantiated with the `create_tracker()` method as follows:

```python
Snowplow.create_tracker(namespace='ns', endpoint='collector.example.com')
```

This creates a `Tracker` and `Emitter` with default settings, with events logged to `collector.example.com`.

You can access a tracker in the following way:

```python
Snowplow.get_tracker('ns')
```

The Snowplow class can be used to initialize trackers using the following properties:

| **Argument Name** | **Description**                      | **Required?** | **Default**              |
| ----------------- | ------------------------------------ | ------------- | ------------------------ |
| `namespace`       | The name of the tracker              | Yes           |                          |
| `endpoint`        | The collector URL events are sent to | Yes           |                          |
| `method`          | The method to use: “get” or “post”   | No            | `post`                   |
| `emitter_config`  | The emitter configuration object     | No            | `EmitterConfiguration()` |
| `app_id`          | The application ID                   | No            | `None`                   |
| `subject`         | The user being tracked               | No            | `subject.Subject()`      |
| `tracker_config`  | The tracker configuration object     | No            | `TrackerConfiguration()` |

Optionally, you may choose to use the `TrackerConfiguration` and `EmitterConfiguration` classes to configure the emitter and tracker respectively.

### Tracker configuration using `TrackerConfiguration`

The `TrackerConfiguration` class contains settings to encode the payload and provide a custom json serializer. For example:

```python
tracker_config = TrackerConfiguration(

    encode_base64=False,

    json_encoder=my_custom_encoder

)



Snowplow.create_tracker(

    namespace='ns',

    endpoint='collector.example.com',

    tracker_config = tracker_config

)
```

| **Argument Name** | **Description**                                                     | **Default** |
| ----------------- | ------------------------------------------------------------------- | ----------- |
| `encode_base64`   | Whether JSONs in the payload should be base-64 encoded.             | `True`      |
| `json_encoder`    | Custom JSON serializer that gets called on non-serializable object. | `None`      |

### Emitter configuration using `EmitterConfiguration`

The `EmitterConfiguration` class contains additional settings for the `Emitter` initialization. For example:

```python
emitter_config = EmitterConfiguration(

    batch_size = 50,

    on_success = my_success_function,

    on_failure= my_failure_function,

    byte_limit = 25000,

    request_timeout = (10, 20),

    custom_retry_codes = {500: False, 401: True}, # Don't retry 500, retry 401

    event_store=my_custom_event_store,

    session=requests.Session

)



Snowplow.create_tracker(

    namespace='ns',

    endpoint='collector.example.com',

    emitter_config = emitter_config

)
```

The full list of arguments is below:

| **Argument Name**    | **Description**                                                                                                                                                                                                                                                               | **Default**          |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `batch_size`         | The maximum number of queued events before the buffer is flushed.                                                                                                                                                                                                             | `10`                 |
| `on_success`         | Callback executed after every HTTP request in a flush has status code 200.                                                                                                                                                                                                    | `None`               |
| `on_failure`         | Callback executed if at least one HTTP request in a flush has status code other than 200.                                                                                                                                                                                     | `None`               |
| `byte_limit`         | The size event list after reaching which queued events will be flushed.                                                                                                                                                                                                       | `None`               |
| `request_timeout`    | Timeout for the HTTP requests. Can be set either as single float value which applies to both "connect" AND "read" timeout, or as tuple with two float values which specify the "connect" and "read" timeouts separately.                                                      | `None`               |
| `custom_retry_codes` | Set custom retry rules for HTTP status codes received in emit responses from the Collector. By default, retry will not occur for status codes 400, 401, 403, 410 or 422. This can be overridden here. Note that 2xx codes will never retry as they are considered successful. | `None`               |
| `event_store`        | Stores the event buffer and buffer capacity.                                                                                                                                                                                                                                  | `InMemoryEventStore` |
| `session`            | Persist parameters across requests by using a session object                                                                                                                                                                                                                  | `None`               |

## Option 2: Managing "Tracker", "Emitter", and "Subject" directly

Require the Python Tracker's module into your Python code like so:

```python
from snowplow_tracker import Tracker, Emitter, Subject
```

That's it - you are now ready to initialize a tracker instance.

### Creating a tracker

The simplest tracker initialization only requires you to provide the Emitter's endpoint to which the tracker will log events:

```python
e = Emitter(endpoint="collector.example.com")

t = Tracker(namespace="snowplow_tracker", emitters=e)
```

The tracker parameters are:

| **Argument Name** | **Description**                                                            | **Required?** | **Default**         |
| ----------------- | -------------------------------------------------------------------------- | ------------- | ------------------- |
| `namespace`       | The name of the tracker instance                                           | Yes           |                     |
| `emitters`        | The emitter(s) to which events are sent                                    | Yes           |                     |
| `subject`         | The user being tracked                                                     | No            | `subject.Subject()` |
| `app_id`          | The application ID                                                         | No            | `None`              |
| `encode_base64`   | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64) | No            | `True`              |
| `json_encoder`    | Custom JSON serializer                                                     | No            | `None`              |

Here is a more complete example in which every tracker parameter is set:

```python
e = Emitter(endpoint="collector.example.com")

s = Subject().set_platform("srv")



tracker = Tracker(

    namespace="cf",

    emitter=e,

    subject=s,

    app_id="aid",

    encode_base64=False,

    json_encoder=my_custom_encoder

)
```

#### `namespace`

The `namespace` argument is attached to every event fired by the new tracker. This allows you to later identify which tracker fired which event if you have multiple trackers running.

#### `emitters`

This can be a single emitter or an array containing at least one emitter. The tracker will send events to these emitters, which will in turn send them to a collector.

```python
e1 = Emitter(endpoint="mycollector.com")

e2 = Emitter(endpoint="myothercollector.com", port=8080)

tracker = Tracker(namespace="snowplow_tracker", emitters=[e1, e2])
```

#### `subject`

The user which the Tracker will track. This should be an instance of the Subject class. You don't need to set this during Tracker construction; in this case the tracker will set a default subject, which you can also change using the `Tracker.set_subject` method afterwards.

#### `app_id`

The `app_id` argument lets you set the application ID to any string.

#### `encode_base64`

By default, unstructured events and custom contexts are encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the Boolean `encode_base64` argument.

#### `json_encoder`

This parameter allows you to customize the JSON encoder used to serialize objects added to the payload. For example:

```python
from json.encoder import JSONEncoder

def complex_encoder(c):

    if isinstance(c,complex):

        return [c.real, c.imag]

    return JSONEncoder.default(c)



t = Tracker(

    namespace="snowplow_tracker",

    emitter=e,

    json_encoder=complex_encoder

)
```

---

# Emitter logs in the Python tracker
> Configure Python logging levels for the tracker emitters module to control debug, info, and error message output.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/logging/

The emitters.py module has Python logging turned to give you information about requests being sent. The logger prints messages about what emitters are doing. By default, only messages with priority "INFO" or higher will be logged.

To change this:

```python
from snowplow_tracker import logger



# Log all messages, even DEBUG messages

logger.setLevel(10)



# Log only messages with priority WARN or higher

logger.setLevel(30)



# Turn off all logging

logger.setLevel(60)
```

---

# Install the Python tracker
> Install the Python tracker from PyPI using pip for event tracking in Python applications with minimal dependencies.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/setup/

As a programming language that lets you work more quickly and integrate your systems more effectively, Python is available in a huge number of different computing environments and platforms, from Civilization IV through [Django framework](https://www.djangoproject.com/) to Ubuntu One.

To make the Snowplow Python Tracker work out-of-the-box with as many different Python programs as possible, we have tried to:

1. Minimize external dependencies and third party libraries
2. Provide setup instructions

## Dependencies

To make the Snowplow Python Tracker work with as many different Python programs as possible, we have tried to keep external dependencies to a minimum. The main external dependencies currently are:

- [Requests](https://pypi.python.org/pypi/requests) – HTTP library
- [Python typing extensions](https://pypi.org/project/typing-extensions/) – Backported type hints for Python

These dependencies can be installed from the package manager of the host system or through PyPi.

## PyPI

The Snowplow Python Tracker is published to [PyPI](https://pypi.python.org/), the the official third-party software repository for the Python programming language.

This makes it easy to either install the tracker locally, or to add it as a dependency into your own Python app.

## pip

To install the Snowplow Python Tracker locally, assuming you already have `pip` installed:

```bash
$ pip install snowplow-tracker --upgrade
```

To add the Snowplow Tracker as a dependency to your own Python app, edit your `requirements.txt` and add:

```txt
snowplow-tracker==1.0.3
```

## Python version support

Please refer to the table below to identify the recommended tracker version for your Python version.

| Python Version | snowplow-tracker Version |
| -------------- | ------------------------ |
| >=3.5          | 1.0.3                    |
| 2.7            | 0.9.1                    |

---

# Track subject data with the Python tracker
> Configure subject data for tracked events including user ID, platform, screen resolution, timezone, and IP address for tracker-level or event-level subjects.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/subject/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

The Subject class has a number of `set_xxx()` methods to attach extra data relating to the user of the tracked events.

Here are some examples:

```python
from snowplow_tracker import Subject

subject = Subject()



subject.set_user_id("user_id")

subject.set_lang("en-gb")

subject.set_screen_resolution(1920, 1080)
```

There are two ways to provide the subject information:

1. For all tracked events. This is useful in client-side applications where all tracked events relate to the same user and share the same properties.
2. For each event individually. This is useful in server-side applications where each event might relate to a different user.

> **Note:** When the same parameter is set for a tracker subject and an event subject, the event subject will take priority.

The full set of subject methods are listed below:

| **Subject Method**                                      | **Description**                                                |
| ------------------------------------------------------- | -------------------------------------------------------------- |
| [`set_platform`](#set_platform)                         | Override the platform for the event (e.g. `srv`, `app`, `web`) |
| [`set_user_id`](#set_user_id)                           | Identify the Subject with a business ID                        |
| [`set_screen_resolution`](#set_screen_resolution)       | Set the dimensions of the Subject's device                     |
| [`set_viewport`](#set_viewport)                         | Set the dimensions of the Subject's viewport                   |
| [`set_color_depth`](#set_color_depth)                   | Set the color depth of the Subject device's display            |
| [`set_timezone`](#set_timezone)                         | Set the local timezone for the Subject                         |
| [`set_lang`](#set_lang)                                 | Set the preferred language or locale of the Subject            |
| [`set_ip_address`](#set_ip_address)                     | Set the IP address of the Subject                              |
| [`set_user_agent`](#set_useragent)                      | Set the User Agent string of the Subject                       |
| [`set_domain_user_id`](#set_domain_user_id)             | Set the Domain User ID of the Subject                          |
| [`set_network_user_id`](#set_network_user_id)           | Set the Network User ID of the Subject                         |
| [`set_domain_session_id`](#set_domain_session_id)       | Set the Domain Session ID of the Subject                       |
| [`set_domain_session_index`](#set_domain_session_index) | Set the Domain Session Index of the Subject                    |

### Setting a Tracker Subject

To configure a tracker subject, pass it to the tracker during initialization:

```python
subject = Subject().set_platform("mob").set_user_id("user-12345").set_lang("en")



t = Tracker(

        namespace="snowplow_tracker",

        emitters=emitter,

        subject=subject

)
```

If you initialize a `Tracker` instance without a subject, a default `Subject` instance will be attached to the tracker. You can access that subject like this:

```python
t = Tracker(

        namespace="snowplow_tracker",

        emitters=my_emitter

)

t.subject.set_platform("mob").set_user_id("user-12345").set_lang("en")
```

### Setting an Event Subject

To configure an event level subject, pass it to the event during initialization.

```python
event_subject = Subject().set_screen_resolution(1920, 1080)



id = tracker.get_uuid()

screen_view = ScreenView(

  id_=id,

  name="name",

  event_subject=event_subject

)



tracker.track(screen_view)
```

### `set_platform`

The default platform is `pc`. You can change the platform the subject is using by calling for example:

```python
s.set_platform('mob')
```

For a full list of supported platforms, please see the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields).

### `set_user_id`

You can set the user ID to any string:

```python
s.set_user_id("user_id")
```

### `set_screen_resolution`

Set the screen resolution as below. Both numbers should be positive integers in the order width followed by height.

```python
s.set_screen_resolution(1366, 768)
```

### `set_viewport`

Set the viewport dimensions as below. Both numbers should be positive integers in the order width followed by height.

```python
s.set_viewport(300, 200)
```

### `set_color_depth`

Set the bit depth of the device's color palette for displaying images as below. The number should be a positive integer, in bits per pixel. For example:

```python
s.set_color_depth(32)
```

### `set_timezone`

This method lets you pass a user's timezone into Snowplow. The timezone should be a [IANA TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) string.

```python
s.set_timezone("Europe/London")
```

### `set_lang`

This method lets you pass a user's language into Snowplow. The language should be a string.

```python
s.set_lang('en')
```

### `set_ip_address`

If you have access to the user's IP address, you can set it like this:

```python
s.set_ip_address('34.633.11.139')
```

The set value will be used in the [IP Lookup](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) and [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) enrichments, if enabled.

> **Tip:** On web applications, the remote address of an incoming request may reflect load balancers or reverse proxies instead of the actual Subject's IP. You may need to consider the [Forwarded](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) or [X-Forwarded-For](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) HTTP headers instead, though these headers can be forged by the requester.

### `set_useragent`

If you have access to the user's useragent (sometimes called "browser string"), you can set it like this:

```python
s.set_useragent('Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0')
```

The set value will be used in the [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/), [UA Parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/), [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) and similar enrichments.

### `set_domain_user_id`

The `domain_userid` field of the Snowplow event model corresponds to the ID stored in the first party cookie set by the Snowplow JavaScript Tracker. If you want to match up server-side events with client-side events, you can set the domain user ID for server-side events like this:

```python
s.set_domain_user_id('c7aadf5c60a5dff9')
```

### `set_network_user_id`

The `network_user_id` field of the Snowplow event model corresponds to the ID stored in the third party cookie set by the Snowplow Collector. You can set the network user ID for server-side events like this:

```python
s.set_network_user_id('ecdff4d0-9175-40ac-a8bb-325c49733607')
```

### `set_domain_session_id`

This method lets you pass a Domain Session ID in to Snowplow:

```python
s.set_domain_session_id('ecdff4d0-9175-40ac-a8bb-325c49733607')
```

> **Tip:** The Domain User ID, Domain Session ID, and Domain Session Index values can all be extracted from the same cookie.

### `set_domain_session_index`

This method lets you pass a Domain Session index in to Snowplow:

```python
s.set_domain_session_index(4)
```

## Tracking multiple subjects

You may want to track more than one subject concurrently. To avoid data about one subject being added to events pertaining to another subject, create two subject instances and switch between them using `Tracker.set_subject`:

```python
from snowplow_tracker import Subject, Emitter, Tracker



# Create a simple Emitter which will log events to https://d3rkrsqld9gmqf.cloudfront.net/com.snowplowanalytics.snowplow/tp2

e = Emitter(endpoint="d3rkrsqld9gmqf.cloudfront.net")



# Create a Tracker instance

t = Tracker(emitters=e, namespace="cf", app_id="CF63A")



# Create a Subject corresponding to a pc user

s1 = Subject()



# Set some data for that user

s1.set_platform("pc")

s1.set_user_id("0a78f2867de")



# Set s1 as the tracker subject

# All events fired will have the information we set about s1 attached

t.set_subject(s1)



# Track user s1 viewing a page

page_view = PageView(

        page_url="https://www.snowplow.io",

        page_title="Homepage",

)

t.track(page_view)



# Create another Subject instance corresponding to a mobile user

s2 = Subject()



# All methods of the Subject class return the Subject instance so methods can be chained:

s2.set_platform("mob").set_user_id("0b08f8be3f1")



# Change the tracker subject from s1 to s2

# All events fired will have instead have information we set about s2 attached

t.set_subject(s2)



# Track user s2 viewing a page

page_view = PageView(

        page_url="https://www.snowplow.io",

        page_title="Homepage",

)

t.track(page_view)



# Switch back to s1 and track a structured event.

t.set_subject(s1)

struct_event = StructuredEvent(

        category="shop",

        action="add-to-basket",

        label="web-shop",

        property_="pcs",

        value=2,

    )

t.track(struct_event)
```

---

# Track specific events with the Python tracker
> Track page views, screen views, structured events, and custom self-describing events with the Python tracker using Event classes and custom context.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/tracking-specific-events/

The Python tracker makes it simple to track a selection of out-of-the-box events as well as the ability to define your own custom events.

To track an event, simply pass the `Event` object to the `tracker.track()` method. For example, tracking a page view:

```python
page_view = PageView(

  page_url="https://www.snowplow.io",

  page_title="Homepage",

)

tracker.track(page_view)
```

Every tracked event payload has a unique event\_id UUID string (`eid`). Other properties include the name\_tracker (`namespace`) and `app_id` set when the Tracker was initialized. From version 1 onwards, `tracker.track()` returns the payload's `eid`.

Snowplow events have a defined structure and protocol that is identical regardless of the tracker used. Further information on this structure can be found [here](/docs/events/).

The Python tracker [Github repository](https://github.com/snowplow/snowplow-python-tracker) includes 3 example apps demonstrating different ways to send events to your collector.

## Event Tracking

The Python tracker provides classes for tracking different types of events. They are listed below:

| **Event**         | **Description**                               |
| ----------------- | --------------------------------------------- |
| `SelfDescribing`  | Track custom events with custom schemas       |
| `PageView`        | Track views of web pages                      |
| `PagePing`        | Track engagement on web pages over time       |
| `ScreenView`      | Track views of a screen (non-web e.g. in-app) |
| `StructuredEvent` | Track custom events without schemas           |

## Creating a custom event (`SelfDescribing`)

To track data using a `SelfDescribing` event, the data must be structured as a `SelfDescribingJson` object. These require two fields, a URI for a self-describing JSON schema and the data in the form of a `PayloadDict`. The data must be valid against the schema.

A simple initialization for a link click event looks like this:

```python
link_click = SelfDescribing(

  SelfDescribingJson(

    "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",

    {"targetUrl": "https://www.snowplow.io"},

  ),

)

tracker.track(link_click)
```

## Creating a `PageView` Event

Track views of a web page with the `PageView` event.

| **Property**     | **Description**                      | **Type**                   | **Required?** |
| ---------------- | ------------------------------------ | -------------------------- | ------------- |
| `page_url`       | URL of the viewed page               | `string`                   | Yes           |
| `page_title`     | Title of the viewed page             | `string`                   | No            |
| `referrer`       | The address which linked to the page | `string`                   | No            |
| `event_subject`  | The subject for the event            | `Subject`                  | No            |
| `context`        | Custom context for the event         | `List(SelfDescribingJson)` | No            |
| `true_timestamp` | When the page view occurred          | `int or float`             | No            |

Example:

```python
page_view = PageView(

  page_url="https://www.snowplow.io",

  page_title="Homepage",

  referrer="https://docs.snowplow.io/docs",

)

tracker.track(page_view)
```

## Creating a `PagePing` Event

Track engagement with a web page over time, via a `PagePing` event. Each ping represents a single heartbeat.

| **Property**     | **Description**                                    | **Type**                   | **Required?** |
| ---------------- | -------------------------------------------------- | -------------------------- | ------------- |
| `page_url`       | URL of the viewed page                             | `string`                   | Yes           |
| `page_title`     | Title of the viewed page                           | `string`                   | No            |
| `referrer`       | The address which linked to the page               | `string`                   | No            |
| `min_x`          | Minimum page x offset seen in the last ping period | `int`                      | No            |
| `max_x`          | Maximum page x offset seen in the last ping period | `int`                      | No            |
| `min_y`          | Minimum page y offset seen in the last ping period | `int`                      | No            |
| `max_y`          | Maximum page y offset seen in the last ping period | `int`                      | No            |
| `event_subject`  | The subject for the event                          | `Subject`                  | No            |
| `context`        | Custom context for the event                       | `List(SelfDescribingJson)` | No            |
| `true_timestamp` | When the page ping occurred                        | `int or float`             | No            |

Example:

```python
page_ping = PagePing(

  page_url="https://www.snowplow.io",

  page_title="Homepage",

  referrer="https://docs.snowplow.io/docs",

)

tracker.track(page_ping)
```

## Creating a `ScreenView` Event

Use the `ScreenView` to track a user viewing a screen (or equivalent) within your app.

| **Property**      | **Description**                                             | **Type**                   | **Required?** |
| ----------------- | ----------------------------------------------------------- | -------------------------- | ------------- |
| `id_`             | Unique identifier for this screen (UUID)                    | `string`                   | No            |
| `name`            | Human-readable name for this screen                         | `Non-empty string`         | No            |
| `type`            | The type of screen that was viewed e.g feed / carousel.     | `string`                   | No            |
| `previous_name`   | The name of the previous screenview.                        | `string`                   | No            |
| `previous_id`     | The id of the previous screenview.                          | `string`                   | No            |
| `previous_type`   | The type of the previous screenview.                        | `string`                   | No            |
| `transition_type` | The type of transition that led to the screen being viewed. | `string`                   | No            |
| `event_subject`   | The subject for the event                                   | `Subject`                  | No            |
| `context`         | Custom context for the event                                | `List(SelfDescribingJson)` | No            |
| `true_timestamp`  | When the screen was viewed                                  | `int or float`             | No            |

Example:

```python
id = tracker.get_uuid()

screen_view = ScreenView(

  id_=id,

  name="name",

  type="feed",

  previous_name="Home Page",

  previous_id="1368725287000",

  previous_type="feed"

)

tracker.track(screen_view)
```

## Creating a `StructuredEvent`

Use `StructuredEvent` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Property**     | **Description**                                                  | **Type**                   | **Required?** |
| ---------------- | ---------------------------------------------------------------- | -------------------------- | ------------- |
| `category`       | The grouping of structured events which this `action` belongs to | `Non-empty string`         | Yes           |
| `action`         | Defines the type of user interaction which this event involves   | `Non-empty string`         | Yes           |
| `label`          | A string to provide additional dimensions to the event data      | `string`                   | No            |
| `property`       | A string describing the object or the action performed on it     | `string`                   | No            |
| `value`          | A value to provide numerical data about the event                | `int or float`             | No            |
| `event_subject`  | The subject for the event                                        | `Subject`                  | No            |
| `context`        | Custom context for the event                                     | `List(SelfDescribingJson)` | No            |
| `true_timestamp` | When the structured event occurred                               | `int or float`             | No            |

Example:

```python
struct_event = StructuredEvent(

  category="shop",

  action="add-to-basket",

  label="web-shop",

  property_="pcs",

  value=2,

)

tracker.track(struct_event)
```

## Common tracking parameters

All events are tracked with specific event classes on the tracker instance, of the form `track(Event)`. The parameters that are common for all track methods are:

- `event_subject`
- `context`
- `true_timestamp`

### Event subject

It is possible to set the Subject per-event, in order to augment the event with extra information without having to change the Subject at the Tracker level. This provides a thread safe way to track multiple subjects.

Event level subjects are combined with any tracker subjects that have been initialized, with the event subject taking priority over tracker subject parameters.

This is supported as an optional keyword argument by all event classes. For example:

```python


event_subject = Subject().set_user_id("1234")

page_view = PageView(

  page_url="https://www.snowplow.io",

  page_title="Homepage",

  event_subject=event_subject

)



tracker.track(page_view)
```

More detail on the `Subject` class can be found [here](/docs/sources/python-tracker/subject/).

### Custom context

Custom context can be used to augment any standard Snowplow event type, including self describing events, with additional data. You can read more about custom contexts and the possible use cases [here](/docs/fundamentals/entities/#custom-entities).

Custom context can be added as an extra argument to any of Snowplow’s Event classes. The `context` argument should consist of a list of one or more instances of the `SelfDescribingJson` class.

For example, if a server-side Python application can determine visitor's geoposition, this can be attached to the event using the `geolocation_context` that is predefined on [Iglu Central](https://github.com/snowplow/iglu-central):

```python
from snowplow_tracker import SelfDescribingJson



geo_context = SelfDescribingJson(

  "iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-0-0",

  {

    "latitude": -23.2,

    "longitude": 43.0

  }

)
```

As another example, if a visitor arrives on a page advertising a movie, the context object might look like this (`movie_poster` is custom context, not predefined):

```python
poster_context = SelfDescribingJson(

  "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",

  {

    "movie_name": "Solaris",

    "poster_country": "JP",

    "poster_year": "1978-01-01"

  }

)
```

This is how to fire a page view event with both above contexts:

```python
page_view = PageView(

  page_url="https://www.snowplow.io",

  page_title="Homepage",

  context=[poster_context, geo_context]

)



tracker.track(page_view)
```

**Important:** Even if only one custom context is being attached to an event, it still needs to be parsed as a list.

### Timestamp argument

Each event class supports an optional timestamp as an argument. The timestamp should be in milliseconds since the Unix epoch, the same format as generated by `time.time() * 1000`.

Generally, according to the Snowplow Tracker Protocol, every event tracked will be recorded with two timestamps:

- the `dvce_created_tstamp`, which is the timestamp when the event was created

- the `dvce_sent_tstamp`, which is the timestamp when the event was sent

These are going to be used downstream, to calculate the `derived_tstamp` for the event, which takes also into account the collector timestamp, in order to best approximate the exact time the event occurred.

The optional timestamp argument is for the cases where you might want to set the event timestamp yourself. If this argument is not provided or set to None, then the Python Tracker will use the current time to be the `dvce_created_tstamp` for the event.

Here is an example tracking a structured event and supplying the optional timestamp argument. We can explicitly supply `None` for the intervening arguments which are empty:

```python
struct_event = StructuredEvent(

  category="shop",

  action="add-to-basket",

  label="web-shop",

  property_="pcs",

  value=2,

  true_timestamp=1368725287000

)

tracker.track(struct_event)
```

---

# Upgrade the Python tracker
> Upgrade to newer versions of the Python tracker with migration guides for v1.0.0 and v0.13.0+ breaking changes and new features.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/upgrading/

This page gives instructions to upgrading to newer versions of the Python Tracker

## Upgrading to v1.0.0

There are a number of breaking changes and new features in the V1 release, in order to make use of the latest tracker make sure to:

- Tracker `namespace` is now a required property in the tracker and should be parsed on initialization of a new tracker.
- The `track_xxx()` functions have been deprecated in favor of the `Event` class, further information on tracking events using the new API can be found on the [Event Tracking](/docs/sources/python-tracker/tracking-specific-events/) page.
- The `track()` function in the tracker now returns an event id, instead of a tracker instance. Chaining methods using the `track()` function will no longer work, and methods should be applied to the tracker itself.
- The `track()` function no longer accepts a `Payload` object as an argument and instead accepts an `Event` object.
- The parameters in the `Tracker` constructor have changed order, to avoid errors we suggest explicitly defining the variables.
- Event subjects and tracker subjects are now combined when an event is tracked on a given tracker. The old API will continue to work, however the contents of the payload in the event will contain additional tracker subject data.
- The `Redis` and `Celery` emitters have now been removed from the Python tracker. An example of a redis implementation can be found in the [examples](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/redis_example) section of the tracker repo.
-

## Upgrading to v0.13.0+

There are a number of breaking changes in this release, in order to make use of the new features please make sure to:

- Use the `batch_size` argument instead of `buffer_size` in the Emitter initialization.
- Users should no longer use the `on_failure` method to determine retry behaviors. Retry failed events is now a built in feature which can be configured in the `Emitter` directly or via `EmitterConfiguration` object.

---

# Remove the React Native tracker at runtime
> Remove individual trackers by namespace or remove all trackers at runtime using the React Native tracker API.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/advanced-usage/

The React Native tracker provides two functions that allow you to remove a tracker (or all of them) at runtime.

### removeTracker

As each tracker is identified by its namespace, in order to remove a tracker at runtime, you need to pass its namespace to the `removeTracker` function.

For example, assuming an existing tracker with namespace `sp1` :

```javascript
import { newTracker, removeTracker } from '@snowplow/react-native-tracker';



// ...



removeTracker('sp1');
```

### removeAllTrackers

The function removeAllTrackers, which accepts no arguments, will remove all trackers created in your app.

```javascript
import { removeAllTrackers } from '@snowplow/react-native-tracker';



removeAllTrackers();
```

---

# Anonymous tracking for the React Native tracker
> Enable anonymous tracking to disable user and session identifiers with client-side and server-side anonymization options.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/anonymous-tracking/

Anonymous tracking is a feature that enables anonymization of various user and session identifiers, to support user privacy when consent for tracking the identifiers isn't given.

On React Native, the following [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) can be anonymized:

- Client-side user identifiers:

  - `user_id`, `domain_userid`, `network_userid`, and `user_ipaddress`, set by you in `Subject`
  - `userId`, a device ID, set automatically in the [session](/docs/sources/web-trackers/tracking-events/session/) entity
  - For mobile only, IDFA identifiers (`appleIdfa` and `appleIdfv` for iOS, `androidIdfa` for Android) in the mobile entity [if configured](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/#platform-context)

- Client-side session identifiers:
  - `previousSessionId` in the session entity, set automatically by the tracker

- Server-side user identifiers:
  - `network_userid` and `user_ipaddress`, set by the [Collector](/docs/pipeline/collector/)

> **Note:** The Collector captures the IP address from the request HTTP headers, and updates the `user_ipaddress` event property. However, if you set the `user_ipaddress` property in the `Subject`, that value has priority.
>
> Similarly, if you set the `network_userid` property in the `Subject`, that value is used instead of the Collector cookie value.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

There are several levels to the anonymization depending on which of the categories are affected.

## Available anonymization options

Because of a change in architecture, the anonymous tracking options for the React Native tracker changed between version 2.x and version 4.x.

| Feature                                         | Version 4.x | Version 1.3 - 2.x |
| ----------------------------------------------- | ----------- | ----------------- |
| Full client-side anonymization                  | ❌           | ✅                 |
| Client-side anonymization with session tracking | ❌           | ✅                 |
| Toggle anonymous tracking at runtime            | ❌           | ✅                 |
| Server-side anonymization                       | ✅           | ✅                 |

## 1. Full client-side anonymization

In this case, you anonymize both client-side user identifiers and client-side session identifiers. This means disabling the session entity altogether.

**Version 4.x:**

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    sessionContext: false, // Session entity won't be added to events

    platformContext: false, // Mobile entity won't be added to events



});
```

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ✅ if set            |
| `domain_userid`     | Atomic                   | ✅ if set            |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌ no mobile entity  |
| `appleIdfv`         | Mobile (platform) entity | ❌ no mobile entity  |
| `androidIdfa`       | Mobile (platform) entity | ❌ no mobile entity  |
| `network_userid`    | Atomic                   | ✅                   |
| `user_ipaddress`    | Atomic                   | ✅                   |

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(

    'appTracker',

    { endpoint: COLLECTOR_URL },

    {

        trackerConfig: {

            sessionContext: false,   // Session entity won't be added to events

            userAnonymisation: true, // Remove user identifiers from Subject and platform entity

        },

    }

);
```

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ no session entity                           |
| `sessionId`         | Session entity           | ❌ no session entity                           |
| `previousSessionId` | Session entity           | ❌ no session entity                           |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

***

## 2. Client-side anonymization with session tracking

This setting disables client-side user identifiers but tracks session information.

**Version 4.x:**

Newer versions of the React Native tracker don't support client-side anonymization with session tracking. Unlike in version 2.x, you can't track the session entity with an anonymized `userId`.

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(

    'appTracker',

    { endpoint: COLLECTOR_URL },

    {

        trackerConfig: {

            sessionContext: true,    // Session entity is tracked

            userAnonymisation: true, // User identifiers anonymized

        },

    }

);
```

With `userAnonymisation: true` and `sessionContext: true`, events include the session entity but the `userId` property is set to a null UUID (`00000000-0000-0000-0000-000000000000`). The IDFA identifiers are also removed from the platform entity.

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ null UUID                                   |
| `sessionId`         | Session entity           | ✅                                             |
| `previousSessionId` | Session entity           | ❌                                             |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

> **Note:** When anonymous tracking is enabled or disabled using `tracker.setUserAnonymisation(true | false)`, the tracker starts a new session. This results in a new `sessionId`.

***

## 3. Server-side anonymization

Server-side anonymization affects user identifiers set by the Collector: the `network_userid` property (set in the server-side cookie) and the user IP address. Use the `serverAnonymization` flag to prevent the Collector from setting these values.

> **Note:** The new API uses the US English spelling "anonymization", rather than the British English "anonymisation" seen in the older API.

**Version 4.x:**

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    serverAnonymization: true,

});
```

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(

    'appTracker',

    { endpoint: COLLECTOR_URL },

    {

        emitterConfig: {

            serverAnonymisation: true,

        },

    }

);
```

***

Setting this flag adds an `SP-Anonymous` HTTP header to requests sent to the Collector. The Snowplow pipeline then anonymizes the `network_userid` and `user_ipaddress` identifiers.

| Identifier          | Location in event        | Included in event?                                  |
| ------------------- | ------------------------ | --------------------------------------------------- |
| `user_id`           | Atomic                   | ✅ if set                                            |
| `domain_userid`     | Atomic                   | ✅ if set                                            |
| `userId`            | Session entity           | ✅                                                   |
| `sessionId`         | Session entity           | ✅                                                   |
| `previousSessionId` | Session entity           | ✅                                                   |
| `appleIdfa`         | Mobile (platform) entity | ✅ if set                                            |
| `appleIdfv`         | Mobile (platform) entity | ✅ if set                                            |
| `androidIdfa`       | Mobile (platform) entity | ✅ if set                                            |
| `network_userid`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |

## 4. Full anonymization

Full anonymization combines client-side and server-side anonymization to remove all user and session identifiers from events.

**Version 4.x:**

Avoid configuring unwanted identifiers for full anonymization in version 4.x of the React Native tracker.

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    sessionContext: false,     // No session entity

    serverAnonymization: true, // Collector won't set network_userid or capture IP

    },

});
```

| Identifier          | Location in event        | Included in event?                                  |
| ------------------- | ------------------------ | --------------------------------------------------- |
| `user_id`           | Atomic                   | ✅ if set                                            |
| `domain_userid`     | Atomic                   | ✅ if set                                            |
| `userId`            | Session entity           | ❌ no session entity                                 |
| `sessionId`         | Session entity           | ❌ no session entity                                 |
| `previousSessionId` | Session entity           | ❌ no session entity                                 |
| `appleIdfa`         | Mobile (platform) entity | ✅ if set                                            |
| `appleIdfv`         | Mobile (platform) entity | ✅ if set                                            |
| `androidIdfa`       | Mobile (platform) entity | ✅ if set                                            |
| `network_userid`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(

    'appTracker',

    { endpoint: COLLECTOR_URL },

    {

        trackerConfig: {

            sessionContext: false,     // Session entity won't be added to events

            userAnonymisation: true,   // Remove user identifiers from Subject and platform entity

        },

        emitterConfig: {

            serverAnonymisation: true, // Collector won't set network_userid or capture IP

        },

    }

);
```

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌                   |
| `appleIdfv`         | Mobile (platform) entity | ❌                   |
| `androidIdfa`       | Mobile (platform) entity | ❌                   |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

***

---

# Track client side properties with the React Native tracker
> Configure and update subject properties including user ID, IP address, timezone, language, and screen resolution at tracker initialization or runtime.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/client-side-properties/

The tracker allows you to add a persistent set of information through the `SubjectConfiguration` which represents the basic information about the user and the app.

- `userId` = null: a custom user identifier.
- `networkUserId` = null: override the network user ID (UUIDv4) assigned by the Collector and stored in cookies.
- `domainUserId` = null: a domain user ID (DUID). When using the [JavaScript tracker](/docs/sources/web-trackers/), this is a generated identifier stored in a first-party cookie. The React Native tracker does not assign it automatically.
- `useragent` = null: a custom user-agent. It overrides the user-agent used by default.
- `ipAddress` = null: an IP address (not automatically set).
- `timezone` (set by the tracker): the current timezone label.
- `language` (set by the tracker): the language set in the device.
- `screenResolution` (set by the tracker): the screen resolution.
- `screenViewPort` = null: the screen viewport size
- `colorDepth` = null: the color depth.

The fields tracked using `SubjectConfiguration` are relevant in client-side tracking. Some are set automatically in all events during enrichment, even when no Subject is added. These properties are marked with `*` below, and discussed below. Timezone, marked with `**`, is only set when a `Subject` is tracked with the event.

Add these fields to an event using `Subject`:

| Property           | Field in raw event | Column(s) in enriched event             |
| ------------------ | ------------------ | --------------------------------------- |
| `userId`           | `uid`              | `user_id`                               |
| `ipAddress`\*      | `ip`               | `user_ipaddress`                        |
| `timezone`\*\*     | `tz`               | `os_timezone`                           |
| `language`         | `lang`             | `br_lang`                               |
| `useragent`\*      | `ua`               | `useragent`                             |
| `viewport`         | `vp`               | `br_viewheight`, `br_viewwidth`         |
| `screenResolution` | `res`              | `dvce_screenheight`, `dvce_screenwidth` |
| `colorDepth`       | `cd`               | `br_colordepth`                         |
| `networkUserId`\*  | `tnuid`            | `network_userid`                        |
| `domainUserId`     | `duid`             | `domain_userid`                         |

As always, be aware of privacy when tracking [personal identifiable information](https://snowplow.io/blog/2020/09/06/user-identification-and-privacy/) such as email addresses or IP addresses. The tracker provides anonymous tracking functionality to mask certain user identifiers. Refer to the [section on anonymous tracking to learn more](/docs/sources/react-native-tracker/anonymous-tracking/).

## Overriding autogenerated event properties

All enriched Snowplow events contain values for `user_ipaddress`, `useragent`, and `network_userid`.

The `user_ipaddress` is automatically added to all enriched events (unless [anonymous tracking with server anonymization](/docs/sources/react-native-tracker/anonymous-tracking/) is enabled). To manually override this, use a `Subject` and set an `ipAddress` string; use an empty string to prevent IP address tracking. Alternatively, use the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/).

The `useragent` is also automatically added but it can be overriden on configuration. Snowplow pipelines provide multiple useragent-parsing [enrichments](/docs/pipeline/enrichments/available-enrichments/). To manually override the detected useragent, use a `Subject` and set a `useragent` string.

The `network_userid` is the cookie value for the event Collector's third-party cookie. The cookie is named `sp` (or `micro` for Snowplow Micro pipelines). To override the Collector cookie’s value with your own generated ID, use a `Subject` object and set `networkUserId`.

The `network_userid` is stored in the tracker and it's kept the same until the app is deleted or the Collector endpoint is changed or the cookie is expired. It is not assigned to events if anonymous tracking with server anonymization is enabled.

## Setting the subject configuration

The client-side properties can be set during tracker initialization using the `newTracker` configuration:

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    // pass the subject properties:

    userId: 'my-user-id',

});
```

See the full [list of options in the configuration section](/docs/sources/react-native-tracker/#configuring-the-tracker).

### Setting the subject data in a tracker instance

It is also possible to set or change the subject properties at runtime, using the `set..` methods of the React Native Tracker. The available methods are:

1. `setUserId`

With this method you can set the userId to a new string. To unset the userId, pass a null value as an argument.

```typescript
tracker.setUserId('newUser');
```

2. `setNetworkUserId`

With this method you can set the `network_userid` to a new string(UUIDv4). To unset, pass a null value as an argument.

```typescript
tracker.setNetworkUserId('44df44bc-8844-4067-9a89-f83c4fe1e62f');
```

3. `setDomainUserId`

With this method you can set the `domain_userid` to a new string(UUIDv4). To unset, pass a null value as an argument.

```typescript
tracker.setDomainUserId('0526be47-32cb-44b2-a9e6-fefeaa5ec6fa');
```

4. `setIpAddress`

With this method you can set the `user_ipaddress` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setIpAddress('123.45.67.89');
```

5. `setUseragent`

With this method you can set the `useragent` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setUseragent('some-useragent-string');
```

6. `setTimezone`

With this method you can set the `os_timezone` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setTimezone('Africa/Cairo');
```

7. `setLanguage`

With this method you can set the `br_lang` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setLanguage('fr');
```

8. `setScreenResolution`

With this method you can set the `dvce_screenwidth` and `dvce_screenheight` fields to new integer values. The argument to this method is an array that represents the ScreenSize as `[width, height]`. For example:

```typescript
tracker.setScreenResolution([123, 456]);
```

9. `setScreenViewport`

With this method you can set the `br_viewwidth` and `br_viewheight` fields to new integer values. The argument to this method is an array that represents the ScreenSize as `[width, height]`. For example:

```typescript
tracker.setScreenViewport([123, 456]);
```

10. `setColorDepth`

With this method you can set the `br_colordepth` to a new value. For example:

```typescript
tracker.setColorDepth(20);
```

Finally, there is an extra "wrapper" method to set may subject properties at once:

- `setSubjectData`

This method accepts as an argument a SubjectConfiguration, with the new values as needed. For example:

```typescript
tracker.setSubjectData({

    userId: 'tester',

    domainUserId: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    language: 'es',

    colorDepth: 50,

    screenResolution: [300, 300],

});
```

---

# Global context for React Native trackers
> Add global context entities to all events using context generators, filter functions, and rulesets with the React Native tracker.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/

**Global context** (also known as global entities) lets you define your own contexts once (e.g. on tracker initialization) and then have this context sent with every single event subsequently recorded in the app. This saves having to manually build and send the context array with every single event fired.

Here is an example that adds a global context entity to all subsequently tracked events:

```typescript
// Create a context entity and add it to global context

let contextEntity = {

  schema: 'iglu:com.acme/user_context/jsonschema/1-0-0',

  data: { userid: 1234, name: 'John Doe' }

};

tracker.addGlobalContexts([contextEntity]);



// The global context will be added to this screen view event as well

tracker.trackScreenViewEvent({ name: 'my-screen-name' });
```

You may not want to add the same context entity to all tracked events. There are several ways to make global context dynamic:

1. Using a **[context generator](#context-generators)** that generates context entities on the fly when events are sent.
2. Using **[filter functions](#filter-functions)** that filter which events to add global context entities to.
3. Using **[rulesets](#rulesets)** that define rules which types of events to accept or reject when generating global context entities.

### Context generators

Generating context on-the-fly is accomplished with **context generators**. A context generator is a callback that will be evaluated with an optional argument that contains useful information.

A sample context generator that conditionally generates a context entity could look like this:

```javascript
const contextGenerator = (args) => {

    if (args.eventType == 'pv') {

        return {

            schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',

            data: { test: 1 },

        };

    }

};

tracker.addGlobalContexts([contextGenerator]);
```

The optional input is an associative array that contains three elements:

- `event` : self-describing JSON
- `eventType` : string
- `eventSchema` : string (schema URI)

Keep in mind that the arguments `eventType` and `eventSchema` are data found in `event`. `eventType` and `eventSchema` are provided for convenience, so that simple tasks don't require users to search through the event payload.

#### `eventType`

This argument is a string taken from the event payload field, `e`.

`eventType` takes the following values:

| Type                         | `e` |
| ---------------------------- | --- |
| Screen view tracking         | ue  |
| Pageview tracking            | pv  |
| Custom structured event      | se  |
| Custom self describing event | ue  |

Further information about the event payload can be found in the [tracker protocol documentation](/docs/events/).

#### `eventSchema`

Users should be aware of the behavior of the argument `eventSchema`. Since 'first-class events' (e.g. structured events, transactions, pageviews, etc.) lack a proper schema (their event type is determined by the `e` field), callbacks will be provided the upper-level schema that defines the payload of all events:

`iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4`

For self describing events, `eventSchema` will be the schema that describes the self describing event, not the event payload. Again, this behavior isn't necessarily uniform, but provides more utility to differentiate events.

### Conditional context providers

We can augment context primitives by allowing them to be sent conditionally. While it's possible to define this functionality within context generators (with conditional logic), conditional context providers simplify common ways of sending contexts that follow certain rules.

The general form is an array of two objects:

`[conditional part, context primitive or [array of primitives]]`

The conditional part is standardized into two options:

- a filter function
- a schema ruleset

### Filter functions

Filter functions take the standard callback arguments defined for context generators, but instead of returning a Self Describing JSON, return a boolean value. As should be expected: `true` will attach the context part, `false` will not attach the context part.

### Example

```javascript
// A filter that will only attach contexts to structured events

function structuredEventFilter(args) {

  return args['eventType'] === 'se';

}

var globalContextDefinition = [structuredEventFilter, contextEntityToBeAdded];

tracker.addGlobalContexts([globalContextDefinition]);
```

### Rulesets

Rulesets define when to attach context primitives based on the event schema. This follows the standard behavior for all callbacks (the schema used to evaluate is the same provided in `eventSchema`, namely the payload schema for "first-class events" and otherwise the schema found within the self describing event).

Here's the specific structure of a ruleset, it's an object with certain optional rules that take the form of fields, each holding an array of strings:

```json
{

    accept: [],

    reject: []

}
```

Some examples, take note that wild-card matching URI path components is defined with an asterisk, `*`, in place of the component:

```javascript
// Only attaches contexts to this one schema

var ruleSetAcceptOne = {

  accept: ['iglu:com.mailchimp/cleaned_email/jsonschema/1-0-0']

};



// Only attaches contexts to these schemas

var ruleSetAcceptTwo = {

  accept: ['iglu:com.mailchimp/cleaned_email/jsonschema/1-0-0',

  'iglu:com.mailchimp/subscribe/jsonschema/1-0-0']

};



// Only attaches contexts to schemas with mailchimp vendor

var ruleSetAcceptVendor = {

  accept: ['iglu:com.mailchimp/*/jsonschema/*-*-*']

};



// Only attaches contexts to schemas that aren't mailchimp vendor

var ruleSetRejectVendor = {

  reject: ['iglu:com.mailchimp/*/jsonschema/*-*-*']

};



// Only attach to Snowplow first class events

var ruleSet = {

  accept: ['iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4']

};
```

You can add a global context entity with a ruleset as follows:

```javascript
var globalContextDefinition = [ruleSet, contextEntityToAdd];

tracker.addGlobalContexts([globalContextDefinition]);
```

### Rule requirements

All rules and schemas follow a standard form:

`protocol:vendor/event_name/format/version`

And rules must meet some requirements to be considered valid:

- Two parts are invariant: protocol and format. They are always `iglu` and `jsonschema` respectively.
  - Wildcards can therefore be used only in `vendor`, `event_name` and `version`.

- Version matching must be specified like so: `*-*-*`, where any part of the versioning can be defined, e.g. `1-*-*`, but only sequential parts are to be wildcarded, e.g. `1-*-1` is invalid but `1-*-*` is valid.

- Vendors require the first two "larger parts":
  - `com.acme.*`

- Vendors cannot be defined with non-wildcarded parts between wildcarded parts:

  - `com.acme.*.marketing.*` is invalid
  - `com.acme.*.*` is valid

### Named global entities

From version 4 onwards, you can alternatively supply the definitions as a mapping object. This associates each global entity with a "name" or "tag" that can be used to reference it in future calls.

For example, in the below we associate a global entity definition with the name `example`, and can then update or remove it by name in later calls.

```javascript
// add the entity defined by `globalContextDefinition` with the name "example"

var globalContextDefinition = [ruleSet, contextEntityToAdd];

tracker.addGlobalContexts({ example: globalContextDefinition });



// update the global entity named "example"; this replaces the entity from `globalContextDefinition`

var globalContextDefinition2 = [ruleSet, differentContextEntity];

tracker.addGlobalContexts({ example: globalContextDefinition2 });



// remove the global entity named "example"; it will no longer appear on subsequent events

tracker.removeGlobalContexts(["example"]);
```

Names can be whatever makes sense to you, but each name can only be associated with a single entity definition at a time (additions with the same name will replace the previous value). The value for the named entity can be the same as unnamed global entities defined with the array method, including generator functions, filter functions, and rulesets and are processed equivalently.

### Global contexts methods

These are the standard methods to add and remove global context entities:

#### `addGlobalContexts`

To add global contexts:

```javascript
// unnamed global entities

tracker.addGlobalContexts([array of global contexts]);



// named global entities

tracker.addGlobalContexts({

  name: globalContext1,

  name2: globalContext2,

});
```

#### `removeGlobalContexts`

To remove a global context:

```javascript
// unnamed global entities

tracker.removeGlobalContexts([array of global contexts])



// named global entities

tracker.removeGlobalContexts(["name", "name2"]) // array of entity names
```

Unnamed global context entities are removed by doing a JSON match of the context to be removed with the added context. So the objects have to be the same in order for them to be matched.

For example:

```javascript
var entity = {

  schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',

  data: { test: 1 },

};

tracker.addGlobalContexts([entity]); // add a global context

tracker.removeGlobalContexts([entity]); // remove the global context
```

#### `clearGlobalContexts`

To remove all named and unnamed global context entities:

```javascript
tracker.clearGlobalContexts();
```

---

# Track custom events with the React Native tracker
> Track self-describing events and attach custom context entities using schemas with the React Native tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/custom-tracking-using-schemas/

Self-describing (self-referential) JSON schemas are at the core of Snowplow tracking. Read more about them [here](/docs/fundamentals/schemas/). They allow you to track completely customised data, and are also used internally throughout Snowplow pipelines.

In all our trackers, self-describing JSON are used in two places. One is in the `SelfDescribing` event type that wraps custom self-describing JSONs for sending. The second use is to attach entities to any tracked event. The entities can describe the context in which the event happen or provide extra information to better describe the event.

## Tracking self-describing events

You may wish to track events which are not directly supported by Snowplow. The solution is [self-describing events](/docs/fundamentals/events/#self-describing-events). Self-describing events are [based on JSON Schemas](/docs/fundamentals/schemas/) and can have arbitrarily many fields.

To define your own custom event, you will need to [create a corresponding schema](/docs/event-studio/data-structures/). Snowplow uses the schema to validate that the JSON containing the event properties is well-formed.

A Self Describing event is a [self-describing JSON](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

**Required properties**

- `schema`: (string) – A valid Iglu schema path. This must point to the location of the custom event’s schema, of the format: `iglu:{vendor}/{name}/{format}/{version}`.
- `data`: (object) – The custom data for your event. This data must conform to the schema specified in the `schema` argument, or the event will fail validation and become a [failed event](/docs/fundamentals/failed-events/).

To track a custom self-describing event, use the `trackSelfDescribingEvent` method of the tracker.

For example, to track a link-click event, which is one whose schema is already published in [Iglu Central](https://github.com/snowplow/iglu-central):

```typescript
tracker.trackSelfDescribingEvent({

    schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',

    data: {targetUrl: 'http://a-target-url.com'}

});
```

### Tracking a custom entity

Custom context can be used to augment any standard Snowplow event type, including self-describing events, with additional data. We refer to this custom context as [entities](/docs/fundamentals/entities/).

The context is an array of entities. More than one entity (of either different or the same type) can be attached to an event. The `context` argument (if it is provided at all) should be a non-empty array.

As with self-describing events, if you want to create your own custom context, you will need to [create a corresponding schema](/docs/event-studio/data-structures/). Snowplow uses the schema to validate that the JSON containing the context properties is well-formed.

Custom contexts can be optionally added as an extra argument to any of the Tracker’s `track..()` methods.

**Note:** Even if only one custom context is being attached to an event, it still needs to be wrapped in an array. Also an empty array is acceptable, which will attach no entities to the event.

For example, a custom context to describe a screen could be:

```typescript
const myScreenContext: EventContext = {

    schema: 'iglu:com.example/screen/jsonschema/1-2-1',

    data: {

        screenType: 'test',

        lastUpdated: '2021-06-11'

    }

};
```

Another example custom context to describe a user on a screen could be:

```typescript
const myUserEntity: EventContext = {

    schema: 'iglu:com.example/user/jsonschema/2-0-0',

    data: {

        userType: 'tester'

    }

};
```

Then, to track, for example, a screenViewEvent with both of these contexts attached:

```typescript
tracker.trackScreenViewEvent(

   { name: 'myScreenName' },

   [ myScreenContext, myUserEntity ]

);
```

It is also possible to add custom contexts globally, so that they are applied to all events within an application. For more information, see [the Global Contexts section](/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/).

---

# Track events from hybrid apps with the React Native tracker
> Track events from WebViews in React Native hybrid apps using event forwarding from the Web tracker or WebView tracker.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/hybrid-apps/

> **Info:** This feature is available since v1.3. To use the Web plugin, you will need v4.2+ of the React Native tracker.

Hybrid apps are mobile apps that in addition to a React Native interface, provide part of the UI through an embedded Web view. Snowplow events are tracked from both the React Native code as well as the Web view. Our goal is to have both events tracked from both places to share the same session and appear as tracked with the same tracker.

## Event forwarding

We recommend using the Web tracker (v4.3+) to forward all Web events to the React Native tracker.

1. Implement the React Native tracker.

2. Implement the [Snowplow Web/JavaScript tracker](/docs/sources/web-trackers/) in the WebView in your app. Make sure to include the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/).

3. Subscribe to WebView event messages.

   ```typescript
   import { getWebViewCallback } from '@snowplow/react-native-tracker';



   const YourWebViewComponent = () => {

       return <WebView

           onMessage={getWebViewCallback()}

           source={{uri: WEB_VIEW_URI}}

           ... />;
   ```

4. Track events as usual.

The Web tracker will automatically intercept all web events and forward them to the React Native tracker. The forwarded events will have the tracker version from Web, e.g. "js-4.1.0", but will otherwise be tracked like the mobile events. They may contain additional information not present in the React Native mobile events, such as a browser useragent string or URL, or Web context entities e.g. the [WebPage entity](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity).

The forwarded events are filtered out of the Web tracker event queue so that they are not tracked twice.

The WebView plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## WebView Tracker

If you don't want to implement a Web tracker in your WebView, you can use the [Snowplow WebView tracker](/docs/sources/webview-tracker/) directly. This could be suitable if you only want to track a small number of events from the WebView. We recommend event forwarding for most use cases.

1. Implement the React Native tracker.

2. Implement the [Snowplow WebView tracker](/docs/sources/webview-tracker/) in the WebView in your app.

3. Subscribe to WebView event messages.

   ```typescript
   import { getWebViewCallback } from '@snowplow/react-native-tracker';



   const YourWebViewComponent = () => {

       return <WebView

           onMessage={getWebViewCallback()}

           source={{uri: WEB_VIEW_URI}}

           ... />;
   ```

4. Manually track [WebView events](/docs/sources/webview-tracker/).

All event types can be tracked with WebView tracker v0.3.0+, but it requires more work than using the event forwarding.

---

# React Native tracker SDK
> Track events in React Native apps for iOS, Android, and Web with the Snowplow tracker SDK supporting Expo and React Navigation.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/

The Snowplow React Native Tracker is purely implemented in JavaScript/TypeScript without the use of native iOS/Android modules. This enables it to support all these platforms and frameworks:

- React Native for mobile and Web
- Expo bare and managed workflow

The current version is 4.8.0.

## Initializing a tracker

The React Native Tracker can be configured with a set of configuration objects. Once you import the module into your app, the `newTracker` function can be used to setup a tracker, which is identified internally by its namespace.

The `newTracker` function can be used to create multiple instances of the tracker in the same app. If you call it using a namespace already used, it will reconfigure the tracker with the same namespace. If you call it with a different namespace, the tracker will create a new independent tracker.

```typescript
import { newTracker } from '@snowplow/react-native-tracker';



const tracker = newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

});
```

## Configuring the tracker

The `newTracker` provides a wide range of optional configuration options that let you configure the autotracking functionality of the tracker, specify how requests to the collector should be constructed and much more. The following snippet shows an overview of all the options and their meaning:

```typescript
const tracker = newTracker({

    // Network configuration:

    // Represents the network communication configuration allowing the tracker to be able to send events to the Snowplow collector.



    // URL of the collector that is going to receive the events tracked by the tracker. The URL can include the schema/protocol (e.g.: `http://collector-url.com`). In case the URL doesn’t include the schema/protocol, the HTTPS protocol is automatically selected.

    endpoint: COLLECTOR_URL,

    // HTTP method to use for the requests ('get' or 'post')

    eventMethod: 'post',

    // A custom path which will be added to the endpoint URL to specify the complete URL of the collector when paired with the POST method.

    postPath: 'com.snowplowanalytics.snowplow/tp2',

    // Custom headers for HTTP requests to the Collector

    customHeaders: {},



    // Tracker configuration:

    // Represents the configuration of the tracker and the core tracker properties indicating what should be tracked in terms of automatic tracking and contexts/entities to attach to the events.



    // Identifies the tracker instance

    namespace: "sp1",

    // Identifies the application

    appId: 'my-app-id',

    // Version of the application (e.g., semver or git commit hash)

    appVersion: '1.0.0',

    // Build number of the application

    appBuild: '1',

    // Platform of the device, allowed options are:

    // - `'mob'`: Mobile/Tablet

    // - `'web'`: Web(including mobile web)

    // - `'pc'`: Desktop/Laptop/Netbook

    // - `'srv'`: Server-side app

    // - `'app'`: General app

    // - `'tv'`: Connected TV

    // - `'cnsl'`: Games Console

    // - `'iot'`: Internet of things

    devicePlatform: 'mob',

    // Whether to base64 encode self-describing event and entity data

    encodeBase64: false,

    // Tracks a deep link entity with the first screen view following a deep link received event

    deepLinkContext: true,

    // Tracks a screen entity referring to the last screen view event

    screenContext: true,

    // Tracks foreground, background events and a lifecycle entity with the app state

    lifecycleAutotracking: true,

    // Tracks an install event on the first start

    installAutotracking: false,

    // Tracks screen_end event and screen_summary entity with information about the screen activity

    screenEngagementAutotracking: true,



    // Platform context configuration:

    // Tracks an entity with information about the device

    platformContext: true,

    // Chooses which optional properties of the platform context to track

    platformContextProperties: [

        PlatformContextProperty.Carrier,

        PlatformContextProperty.Language

    ],

    // Provides custom functions to retrieve the platform context information

    platformContextRetriever: {

        getAppAvailableMemory: () => Promise.resolve(1024),

    },



    // Tracker plugins – An array of plugins to be used by the tracker

    plugins: [SnowplowEcommercePlugin()],



    // Session configuration:

    // Represents the configuration of the Session context which gets attached to each event tracked and changes based on the timeout set for the inactivity of app when in foreground or background.



    // Whether to track session information

    sessionContext: true,

    // Timeout for the session when app is in foreground

    foregroundSessionTimeout: 1800,

    // Timeout for the session when app is in background

    backgroundSessionTimeout: 1800,



    // Emitter configuration:

    // Represents the tracker configuration from the emission perspective. It can be used to setup details about how the tracker should treat the events once they have been processed but not yet sent.



    // Number of events to buffer before making a request to collector

    bufferSize: 1,

    // Max size of POST requests

    maxPostBytes: 40000,

    // Max size of GET requests

    maxGetBytes: 40000,

    // Whether to anonymise server-side user identifiers including the `network_userid` and `user_ipaddress`

    serverAnonymization: false,

    // The maximum amount of events that will be buffered in the event store

    maxEventStoreSize: 1000,

    // Whether to use AsyncStorage for event store persistence or an in-memory event store

    useAsyncStorageForEventStore: true,



    // Subject configuration:

    // Represents the configuration of the subject. The SubjectConfiguration can be used to provide the basic information about the user.



    // The custom user identifier. Commonly used for user self-identification – for example after sign in

    userId: 'my-user-id',

    // Override the `network_userid` field assigned by collector. Should contain a valid UUID4 string.

    networkUserId: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    // Populates the `domain_userid` field, which is a user identifier generated and tracked by the JavaScript tracker on Web. Should contain a valid UUID4 string.

    domainUserId: '7cdd5ea8-b0f5-47ea-a8bb-5ec8e98cdbd6',

    // The custom useragent. It populates the `useragent` field.

    useragent: 'some-useragent-string',

    // The IP address of the user. It populates the `user_ipaddress` field.

    ipAddress: '123.45.67.89',

    // The current timezone label. Populates the `os_timezone` field.

    timezone: 'Europe/London',

    // The language set in the device.

    language: 'en',

    // The screen resolution. The screen resolution size can be set as an array of [width, height]. Populates the event fields `dvce_screenwidth` and `dvce_screenheight`.

    screenResolution: [123, 456],

    // The screen viewport. The screen viewport size can be set as an array of [width, height]. Populates the event fields `br_viewwidth` and `br_viewheight`.

    screenViewport: [123, 456],

    // The color depth. Populates the `br_colordepth` field.

    colorDepth: 20,

});
```

To learn more about the specific parts of the configuration and autotracked information, see:

- [Platform and application context entities.](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/)
- [Session tracking.](/docs/sources/react-native-tracker/tracking-events/session-tracking/)
- [App lifecycle tracking.](/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/)
- [Screen tracking.](/docs/sources/react-native-tracker/tracking-events/screen-tracking/)
- [Installation tracking.](/docs/sources/react-native-tracker/tracking-events/installation-tracking/)

## Tracker methods

### Tracking events

The React Native Tracker can track a range of out-of-the-box events. As an example, the following snippet tracks a screen view event:

```ts
tracker.trackScreenViewEvent({

    name: 'my-screen-name',

    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    type: 'carousel',

    transitionType: 'basic'

});
```

The available events are:

- [Self-describing events.](/docs/sources/react-native-tracker/custom-tracking-using-schemas/)
- [Structured events.](/docs/sources/react-native-tracker/tracking-events/#tracking-structured-events)
- [Screen view events.](/docs/sources/react-native-tracker/tracking-events/#tracking-screen-view-events)
- [Page view events.](/docs/sources/react-native-tracker/tracking-events/#tracking-page-view-events)
- [Timing events.](/docs/sources/react-native-tracker/tracking-events/#tracking-timing-events)

You can find out about all available track methods with details and examples in the [Tracking events](/docs/sources/react-native-tracker/tracking-events/) section that follows.

### Adding or removing global contexts

Global context lets you define your own context entities once and then have them sent with every single event subsequently recorded in the app. This saves having to manually build and send the context array with every single event fired and enables you to add entities to autotracked events.

The tracker methods to do so are:

- **addGlobalContexts**
- **removeGlobalContexts**
- **clearGlobalContexts**

You can find out more information and examples about [how to add or remove global contexts here](/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/).

### Setting the Subject

It is possible to change subject data at runtime, as the user journey evolves. Once initialized, the React Native tracker provides the methods to do so for each subject property:

- **setUserId**
- **setNetworkUserId**
- **setDomainUserId**
- **setIpAddress**
- **setUseragent**
- **setTimezone**
- **setLanguage**
- **setScreenResolution**
- **setScreenViewport**
- **setColorDepth**

Alternatively, you can use the **setSubjectData** method to provide a new SubjectConfiguration.

You can find out more on how to use the tracker's methods to [set the subject information at runtime here](/docs/sources/react-native-tracker/client-side-properties/).

### Getting session context data from the tracker

You can also get back session data from the tracker at runtime, that may be useful for your tracking and data modeling design. The available methods are:

- **getSessionUserId**: To get the session user identifier.
- **getSessionId**: To get the session identifier.
- **getSessionIndex**: To get the session index.
- **getIsInBackground**: To get whether the app is currently in background state or not.
- **getBackgroundIndex**: To get the number of background transitions in the current session.
- **getForegroundIndex**: To get the number of foreground transitions in the current session.

You can find out more on how to use the tracker's methods to get session data at runtime in the corresponding [session tracking documentation](/docs/sources/react-native-tracker/tracking-events/session-tracking/#getting-session-data-from-the-tracker).

## Removing a tracker

The React Native Tracker API also provides functions to remove a tracker or remove all trackers at runtime. You can find out how in the [Removing trackers at runtime](/docs/sources/react-native-tracker/advanced-usage/) documentation.

---

# Migration guides for the React Native tracker
> Upgrade guides for React Native tracker versions with breaking changes and migration instructions.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/

This section contains information to help you upgrade to newer versions of the React Native tracker.

---

# Migrate from v0 to v1
> Migration guide for upgrading React Native tracker from v0.2.0 to v1.0.0 with API changes and configuration updates.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/migrating-from-v0-x-to-v1/

This sections describes the differences between v0.2.0 and [v1.0.0](/docs/sources/react-native-tracker/) of the Snowplow React Native Tracker and the steps needed to upgrade, which is also recommended.

The v1 introduces a new API for initializing and configuring a tracker, which is a breaking change from v0. There are also few more changes to consider, for which you will find a dedicated section below.

In the following sections we assume a starting version of 0.2.0. If you have instrumented a tracker version prior to v0.2.0, you can begin from the [React Native Tracker v0 reference](/docs/sources/react-native-tracker/previous-version/react-native-tracker-v0-reference/) page first.

## Tracker initialization

In the v1 of the React Native Tracker, a tracker is configured with a set of configuration objects, about which you can find detailed information [here](/docs/sources/react-native-tracker/).

The `createTracker` function now accepts 3 arguments:

1. **Required** - The tracker namespace
2. **Required** - The NetworkConfiguration
3. _Optional_ - The TrackerControllerConfiguration

In the following examples, the changes needed are described:

### Required arguments

**v0.2.0**

```typescript
import { createTracker } from '@snowplow/react-native-tracker';



const tracker = createTracker(

    'my-tracker-namespace',

    {

        endpoint: 'my-endpoint.com',

        appId: 'my-app-id'

    }

);
```

**v1.0.0**

```typescript
import { createTracker } from '@snowplow/react-native-tracker';



const tracker = createTracker(

    'my-tracker-namespace',

    {

      endpoint: 'my-endpoint.com',

    },

    {

        trackerConfig: {

            appId: 'my-app-id'

        }

    }

);
```

**Note**: The `appId` is **not** a required argument for v1. It is only included in the examples, since it was a required argument for v0.2.0.

### Endpoint

In v0.2.0, the collector URL cannot include the schema/protocol, which can be configured using the `protocol` property (which defaults to HTTPS).

In v1, the `protocol` property has been removed. The collector URL can include the schema/protocol (e.g.: `http://collector-url.com`). In case the URL doesn't include the schema/protocol, the HTTPS protocol is automatically selected.

**v0.2.0 - example http**

```typescript
const tracker = createTracker(

    'my-tracker-namespace',

    {

        endpoint: 'my-endpoint.com',

        protocol: 'http',

        appId: 'my-app-id'

    }

);
```

**v1.0.0 - example http**

```typescript
const tracker = createTracker(

    'my-tracker-namespace',

    {

      endpoint: 'http://my-endpoint.com',

    },

    {

        trackerConfig: {

            appId: 'my-app-id'

        }

    }

);
```

### Initialization options

The tracker initialization options in v0.2.0 are fairly limited compared to the ones [available for v1](/docs/sources/react-native-tracker/quick-start-guide/#instrumentation).

The following example depicts how to map the configuration options of v0.2.0 to v1 configuration.

**v0.2.0**

```typescript
const tracker = createTracker('my-namespace', {

    endpoint: 'my-endpoint.com',

    appId: 'my-app-id',

    method: 'post',

    base64Encoded: true,     // (a)

    platformContext: true,

    applicationContext: true,

    sessionContext: true,

    screenContext: true,

    foregroundTimeout: 600,  // (b)

    backgroundTimeout: 300,  // (c)

    checkInterval: 15,       // (d)

    lifecycleEvents: true,   // (e)

    installTracking: true    // (f)

  }

);
```

**v1.0.0**

```typescript
const tracker = createTracker(

    'my-namespace',

    {

        endpoint: 'my-endpoint.com'

        method: 'post'

    },

    {

        trackerConfig: {

            appId: 'my-app-id',

            base64Encoding: true,    // (a)

            platformContext: true,

            applicationContext: true,

            sessionContext: true,

            screenContext: true,

            lifecycleAutotracking: false,  // (e)

            installAutotracking: true,     // (f)

        },

        sessionConfig: {

            foregroundTimeout: 1800,  // (b)

            backgroundTimeout: 1800   // (c)

        }

    }

);
```

- **(a)** - Renamed `base64Encoded` to `base64Encoding`, which becomes a `TrackerConfig` property
- **(b)** - Default value change for `foregroundTimeout`, which becomes a `SessionConfig` property
- **(c)** - Default value change for `backgroundTimeout`, which becomes a `SessionConfig` property
- **(d)** - `checkInterval` is removed
- **(e)** - Renamed `lifecycleEvents` to `lifecycleAutotracking`
- **(f)** - Renamed `installTracking` to `installAutotracking`

## Tracking events

Generally, the tracking methods of v1 continue to have the same argument logic:

```typescript
// pseudocode

tracker.track..(eventData, eventContexts);
```

The differences are about the eventData properties, where they now match the corresponding schema properties. This affects:

### `trackScreenViewEvent`

**v0.2.0**

```typescript
tracker.trackScreenViewEvent({

  screenName: 'my-screen-name',   // (a)

  screenType: 'carousel',         // (b)

  transitionType: 'basic'

});
```

**v1.0.0**

```typescript
tracker.trackScreenViewEvent({

    name: 'my-screen-name',                      // (a)

    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    type: 'carousel',                            // (b)

    previousName: 'previous-screen',

    previousId: '00d71340-342e-4f3d-b9fd-4de728ffba7a',

    previousType: 'feed',

    transitionType: 'basic'

});
```

Apart from the fact that more ScreenViewEvent properties are available, the changes to v0.2.0 ScreenViewEvent properties are:

- **(a)** - `screenName` becomes `name`
- **(b)** - `screenType` becomes `type`

### `trackPageViewEvent`

**v0.2.0**

```typescript
tracker.trackPageViewEvent({

  pageUrl: 'https://my-url.com',

  pageTitle: 'My page title',

  pageReferrer: 'http://some-other-url.com'  // (a)

});
```

**v1.0.0**

```typescript
tracker.trackPageViewEvent({

  pageUrl: 'https://my-url.com',

  pageTitle: 'My page title',

  referrer: 'http://some-other-url.com'      // (a)

});
```

The changes to v0.2.0 PageViewEvent properties are:

- **(a)** - `pageReferrer` becomes `referrer`

## Setting the subject

In v1.0.0, setting the subject can be done both when configuring the tracker (through the [SubjectConfiguration](/docs/sources/react-native-tracker/previous-version/react-native-tracker-v2-reference/introduction/#configuring-the-tracker) object) and at runtime ([using the `set..` tracker methods](/docs/sources/react-native-tracker/previous-version/react-native-tracker-v2-reference/client-side-properties/#setting-the-subject-data-in-a-tracker-instance)), so the exact migration steps depend on the specifics of your app.

A notable difference between the SubjectConfiguration properties of v1.0.0 from the `SubjectData` properties of v0.2.0 is the way to set screen dimensions either for the screen resolution or for the screen viewport.

**v0.2.0**

```typescript
tracker.setSubjectData({

    screenWidth: 111,

    screenHeight: 222,

    viewportWidth: 333,

    viewportHeight: 444

});
```

**v1.0.0**

```typescript
tracker.setSubjectData({

    screenResolution: [111, 222],

    screenViewport: [333, 444]

});



// or, through more specific methods

tracker.setScreenResolution([111, 222]);

tracker.setScreenViewport([333, 444]);
```

In v1.0.0, the screen dimensions are not specified separately. Instead, the ScreenSize is represented through an array: `[width, height]`.

## Installation considerations for iOS

Because of a version mismatch [issue](https://github.com/snowplow-incubator/snowplow-react-native-tracker/issues/110) that existed in v0.2.0 between the package version and the podspec version of the React Native Tracker, the following steps are needed in order to make sure the RNSnowplowTracker Pod references the new v1.0.0.

```bash
# remove olds Pods and Podfile.lock

rm -rf Pods

rm Podfile.lock



# clean the caches

rm -rf ~/Library/Caches/CocoaPods

rm -rf ~/Library/Developer/Xcode/DerivedData/*



# deintegrate

pod deintegrate



# setup

pod setup



# install pods

pod install
```

---

# Migrate from v1 to v2
> Migration guide for upgrading React Native tracker from v1 to v2 with installation changes, iOS platform updates, and automatic tracking configuration.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/migrating-from-v1-x-to-v2/

Version 2 of the React Native tracker underwent a large rewrite of the internal code, both on the side of the React Native project as well as in the underlying iOS and Android trackers (which have been upgraded to version 5). However, the public APIs and tracker behavior have remained largely unchanged except for a few things discussed on this page.

## Installation on iOS

In addition to adding the tracker as a dependency in your `package.json`, you will also need to add the FMDB dependency to you `ios/Podfile` (unless using Expo Go) with `modular_headers` enabled. Add this line to the end of the file:

```rb
pod 'FMDB', :modular_headers => true
```

## Supported iOS platforms

The supported platforms have changed on iOS:

- Minimum iOS deployment target changed from 9.0 to 11.0.
- On macOS, from 10.10 to 10.13.
- On tvOS, from 9.0 to 12.0.
- On watchOS, from 2.0 to 6.0.

## IDFA tracking on iOS

The approach to track the IDFA identifier has changed in the underlying iOS tracker version 5. This means that you will need to take a different approach to set it up. The new approach [is described in the documentation here](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/).

## Automatic screen view tracking default

The automatic screen view tracking configuration (set by `TrackerConfiguration.screenViewAutotracking`) has been changed to disabled by default. In case you rely on the default value and want to keep using the feature, please set it manually when creating a new tracker.

The feature was disabled because it does not track screen views for React Native screens but for UIKit and Android Activity screens which is not what users expected in most cases.

---

# Migrate from v2 to v4
> Migration guide for upgrading React Native tracker from v2 to v4 with JavaScript-only architecture, new platform support, and configuration changes.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/migrating-from-v2-x-to-v4/

Version 4 of the React Native tracker brings a significant rearchitecture of the tracker, which is now JavaScript-only instead of building on top of the iOS and Android native trackers. This brings support for new platforms (Web, Expo Go) and new features (JS tracker plugins, improved global context), but also brings some limitations.

### Initialize tracker instance using `newTracker` instead of `createTracker`

The method for creating a new tracker instance has been changed to `newTracker`. This is not just a rename, but a change in the structure of its parameters, which are now more in line with the rest of our JavaScript trackers.

You can review the [documentation page](/docs/sources/react-native-tracker/) for the current set of configuration options.

The parameters are now provided in a flattened object instead of separated under categories. Some have also been renamed:

- `method` to `eventMethod`
- `requestHeaders` to `customHeaders`
- `base64Encoding` to `encodeBase64`
- `serverAnonymisation` to `serverAnonymization`
- `customPostPath` to `postPath`

The `logLevel` parameter has been removed. Also global context configuration has been moved outside of the configuration, see [the global context docs page](/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/).

### User anonymization not available yet

The `userAnonymisation` configuration option has been removed. We will reintroduce it in the upcoming versions of the React Native tracker.

The session context entity can still be removed using the `sessionContext` option and one can disable the user identifiers in the platform context and subject properties individually. Read more about the options [in the documentation here](/docs/sources/react-native-tracker/anonymous-tracking/).

### GDPR configuration not available

We have deprecated and removed the GPDR configuration. As a more up-to-date replacement, one can make use of the [consent plugins available for the JavaScript tracker](/docs/sources/web-trackers/tracking-events/consent-gdpr/).

### Application context is not tracked automatically

The application context entity with the current app version and build number is not tracked automatically anymore, but the version information can be provided in the `newTracker` call in which case the tracker will track this entity with events. Read more [in the documentation here](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/).

### Removed geolocation context entity

The geolocation context entity is no longer available to be tracked by the tracker. You may track it manually using global context and [the geolocation schema](/docs/events/ootb-data/geolocation/#geolocation-entity).

### Removed exception and diagnostic autotracking

The exception and diagnostic autotracking features have been deprecated and removed.

### Removed screen view autotracking for UIKit and Android Activities

Screen view autotracking for iOS UIKit and Android Activity views has been deprecated and removed as it is not relevant in most React Native apps. See the [documentation for screen view tracking options](/docs/sources/react-native-tracker/tracking-events/screen-tracking/).

### Less autotracked information in the platform context entity

There are fewer automatically tracked properties in the platform context entity, but it's possible to provide values for them manually. Refer to [the documentation to learn more](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/).

### Not possible to access from native code

Since the tracker is now purely implemented in JavaScript, it is no longer possible to access it from mobile native iOS or Android code.

### Base64 encoding disabled by default

In v2, the tracker defaulted to base64-encoding payloads for Self-Describing events and Entities. In v4, the default is now to not do this. In most cases, this behavior is better for performance and has no other impacts.

In some pipeline architectures, the Collector is protected by a Web Application Firewall; if your custom data contains content that appears risky to the WAF (e.g. HTML content) the removal of base64 encoding may mean those events now get blocked by the firewall.

You can return to the old behavior by specifying `encodeBase64: true` when calling `newTracker` as described above.

---

# React Native tracker quick start guide
> Install and initialize the React Native tracker using npm, configure the endpoint, and start tracking events in your mobile app.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/quick-start-guide/

In order to start sending events using the Snowplow React Native Tracker, the following steps are involved:

## Installation

To install the tracker, add it as a dependency to your React Native app:

```bash
npm install --save @snowplow/react-native-tracker
```

Note that in addition to `react` and `react-native`, the tracker has two peer dependencies that need to be added as dependencies to your app:

1. `@react-native-async-storage/async-storage` for event and session local storage implementation.
2. `react-native-get-random-values` as a polyfill for the Crypto Web APIs.

## Instrumentation

Next, in your app create a new tracker using the `newTracker` method. As a minimal example:

```typescript
import { newTracker } from '@snowplow/react-native-tracker';



const tracker = newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

});
```

The `newTracker` function takes the tracker configuration as the only argument. There are two required properties within the configuration:

1. The tracker namespace, which identifies each tracker instance.
2. The collector URL endpoint.

## Track events

Once the tracker is initialized, you can use the tracker methods to track events, about which you can find out more in the following Tracking events section. As an example, you can track a screen view event like this:

```ts
tracker.trackScreenViewEvent({

    name: 'my-screen-name',

    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    type: 'carousel',

    transitionType: 'basic'

});
```

---

# Track out-of-the-box events with the React Native tracker
> Track screen views, structured events, page views, timing events, and ecommerce transactions with the React Native tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/

The React Native tracker captures two types of out-of-the-box events, automatically captured and manual events.

## Auto-tracked events and entities

Many of the automatic tracking options available on iOS and Android are also available in React Native – these can be enabled or disabled in the TrackerConfiguration passed to `newTracker` as part of the TrackerController configuration object:

```typescript
const tracker = newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,



    // auto-tracked events:

    lifecycleAutotracking: false,

    installAutotracking: true,

    screenEngagementAutotracking: true,



    // auto-tracked context entities:

    applicationContext: true,

    platformContext: true,

    sessionContext: true,

    deepLinkContext: true,

    screenContext: true,

);
```

The automatically captured data are:

- [**Platform and Application Context Tracking**](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/): Captures contextual information about the device and the app.
- [**Session Tracking**](/docs/sources/react-native-tracker/tracking-events/session-tracking/): Captures the session which helps to keep track of the user activity in the app.
- [**App Lifecycle Tracking**](/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/): Captures application lifecycle state changes (foreground/background transitions).
- [**Screen View and Engagement Tracking**](/docs/sources/react-native-tracker/tracking-events/screen-tracking/): Captures each time a new “screen” is loaded.
- [**Installation Tracking**](/docs/sources/react-native-tracker/tracking-events/installation-tracking/): Captures an install event which occurs the first time an application is opened.

## Manually-tracked events

All tracker's `track` methods take two arguments: An object of key-value pairs for the event’s properties, and an optional array of [custom event contexts](/docs/sources/react-native-tracker/custom-tracking-using-schemas/).

### Tracking screen view events

Track the user viewing a screen within the application.

To track a ScreenViewEvent, use the `trackScreenViewEvent` tracker method. For example:

```typescript
tracker.trackScreenViewEvent({

    name: 'my-screen-name',

    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    type: 'carousel',

    transitionType: 'basic'

});
```

The tracker will automatically assign references to the previously tracked screen view event in the `previousName`, `previousId`, and `previousType` properties.

**Required properties**

- `name`: (string) - The name of the screen viewed

**Optional properties**

- `id`: (string) - The id(UUID) of screen that was viewed
- `type`: (string) - The type of screen that was viewed
- `previousName`: (string) - The name of the previous screen that was viewed (assigned automatically)
- `previousId`: (string) - The id(UUID) of the previous screen that was viewed (assigned automatically)
- `previousType`: (string) - The type of screen that was viewed (assigned automatically)
- `transitionType`: (string) - The type of transition that led to the screen being viewed

### Tracking page view events

Track a page view event with the `trackPageViewEvent()` method. Typically this is uncommon in apps, but is sometimes used where fitting data into an existing page views model is required. To track page views from an in-app browser, it is advisable to use the javascript tracker in-browser.

An example:

```typescript
tracker.trackPageViewEvent({

  pageUrl: 'https://my-url.com',

  pageTitle: 'My page title',

  referrer: 'http://some-other-url.com'

});
```

Required properties

- `pageUrl`: (string) – Page Url for the page view event. Must be a valid url.

Optional properties

- `pageTitle`: (string) – Page Title for the page view event.
- `referrer`: (string) – Url for the referring page to the page view event. Must be a vaild url.

### Tracking timing events

Use the `trackTimingEvent` tracker method to track user timing events such as how long resources take to load.

For example:

```typescript
tracker.trackTimingEvent({

    category: 'timing-category',

    variable: 'timing-variable',

    timing: 5,

    label: 'optional-label'

});
```

**Required properties**

- `category`: (string) - Defines the timing category
- `variable`: (string) - Define the timing variable measured
- `timing`: (number) - Represent the time

**Optional properties**

- `label`: An optional string to further identify the timing event

### Tracking consent granted events

Use the `trackConsentGrantedEvent` method to track a user opting into data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied.

For example:

```typescript
tracker.trackConsentGrantedEvent({

    expiry: '2022-01-01T00:00:00Z',

    documentId: 'doc-id',

    version: '1.2',

    name: 'doc-name',

    documentDescription: 'consent doc description'

});
```

**Required properties**

- `expiry`: (string) - The expiry (date-time string, e.g.: '2022-01-01T00:00:00Z')
- `documentId`: (string) - The consent document id
- `version`: (string) - The consent document version

**Optional properties**

- `name`: (string) - The consent document name
- `documentDescription`: (string) - The consent document description

### Tracking consent withdrawn events

Use the `trackConsentWithdrawnEvent` method to track a user withdrawing consent for data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied. To specify that a user opts out of all data collection, `all` should be set to `true`.

For example:

```typescript
tracker.trackConsentWithdrawnEvent({

    all: true,

    documentId: 'doc-id',

    version: '1.2',

    name: 'doc-name',

    documentDescription: 'consent doc description'

});
```

**Required properties**

- `all`: (boolean) - Whether user opts out of all data collection
- `documentId`: (string) - The consent document id
- `version`: (string) - The consent document version

**Optional properties**

- `name`: (string) - The consent document name
- `documentDescription`: (string) - The consent document description

### Tracking ecommerce transaction events

Modelled on Google Analytics ecommerce tracking capability, an ecommerce-transaction event can be tracked as follows:

1. **Create a EcommerceTransaction event object**. This will be the object that is loaded with all the properties relevant to the specific transaction that is being tracked including all the items (see `EcommerceItem` right below) in the order, the prices of the items, the price of shipping and the `order_id`.
2. **Track the transaction** using the `trackEcommerceTransaction` method.

The procedured outlined above will result in the following events being tracked:

1. An EcommerceTransaction event
2. As many as EcommerceItem events as the items passed to the EcommerceTransaction object, that will also inherit the `orderId` from the parent transaction event

#### EcommerceItem

More specifically, to start with, the properties of an `EcommerceItem` are:

**Required properties**

- `sku`: (string) - The item sku
- `price`: (number) - The item price
- `quantity`: (number) - The quantity purchased

**Optional properties**

- `name`: (string) - The item name
- `category`: (string) - The item category
- `currency`: (string) - The item currency

For example:

```typescript
const ecomItem: EcommerceItem = {

    sku: 'DD44',

    name: 'T-Shirt',

    category: 'Green Medium',

    price: 15,

    quantity: 1,

    currency: 'USD'

};
```

#### EcommerceTransaction

An ecommerce transaction object has the following properties:

**Required properties**

- `orderId`: (string) - The order ID of the transaction
- `totalValue`: (number) - The total value of the transaction
- `items`: (array of `EcommerceItem`) - The ecommerce items purchased in the transaction.

**Optional properties**

- `affiliation`: (string)
- `taxValue`: (number)
- `shipping`: (number)
- `city`: (string)
- `state`: (string)
- `country`: (string)
- `currency`: (string)

```typescript
const ecomTransaction: EcommerceTransactionProps = {

    orderId: '1234',

    totalValue: 15,

    items: [ ecomItem ],

    affiliation: 'Womens Apparel',

    taxValue: 1.5,

    shipping: 2.99,

    city: 'San Jose',

    state: 'California',

    country: 'USA',

    currency: 'USD'

};
```

#### `trackEcommerceTransaction`

Then, to track the ecommerce transaction as described in the above examples:

```typescript
tracker.trackEcommerceTransactionEvent(ecomTransaction);
```

### Tracking deep link received events

The Deep Link is received by the mobile operating system and passed to the related app. Our React Native tracker can't automatically track the Deep Link, but we provide an out-of-the-box event that can be used by the developer to manually track it as soon as the Deep Link is received in the app.

It will be the duty of the tracker to automatically attach the information of the Deep Link to the first Screen View tracked.

In practice, when the app receives a Deep Link the developer can track it through the `trackDeepLinkReceivedEvent` endpoint:

```typescript
tracker.trackDeepLinkReceivedEvent({

    url: 'https://deeplink.com',

    referrer: 'http://refr.com',

});
```

Please refer to the [Linking API](https://reactnative.dev/docs/linking) in React Native for information on how to retrieve the linked URLs.

The tracker keeps memory of the tracked Deep Link event and will attach a Deep Link entity to the first ScreenView tracked in the tracker. This is helpful during the analysis of the data because it will be clear the relation between the content visualized by the user (ScreenView event) and source (DeepLink entity) that originated that visualisation.

This behavior is enabled by default but it can be disabled using the `deepLinkContext` boolean property when passing `trackerConfig` to `createTracker`:

```typescript
const tracker = createTracker(

    namespace, networkConfig,

    {

        trackerConfig: {

            deepLinkContext: false

        }

    }

);
```

The Deep Link Received event can be used in pair with a campaign-attribution-enrichment appropriately enabled in the Snowplow pipeline. It works exactly like for Page View events in the web/JS tracker. When the user taps on an advertising banner or a marketing email or message, it can trigger the launch of the app through the Deep Linking feature. The referral from the advertising campaigns, websites, or other source can be composed by UTM parameters used to attribute the user activity back to the campaign. The Campaign Attribution Enrichment can parse the DeepLinkReceived event extracting the UTM parameters in the deep link url.

Required properties

- `url`: (string) – URL in the received deep-link

Optional properties

- `referrer`: (string) – Referrer URL, source of this deep-link

### Tracking push and local notification events

Push Notifications are a cornerstone of the user experience on mobile. A Push Notification is a message (like an SMS or a mobile alert) that can quickly show information to the user even if an app is closed or the phone is in stand-by. Push Notifications can be used in many different ways: they can notify of a new message in a chat, rather than informing of a news or just notify of an achievement in a fitness app.

To track an event when a push (or local) notification is used, it is possible to use the `trackMessageNotificationEvent` endpoint:

```typescript
tracker.trackMessageNotificationEvent({

    title: 'title',

    body: 'body',

    trigger: 'push',

    action: 'action',

    attachments: [

        {

            identifier: 'att_id',

            type: 'att_type',

            url: 'http://att.url',

        },

    ],

    bodyLocArgs: ['bodyArg1', 'bodyArg2'],

    bodyLocKey: 'bodyKey',

    category: 'category',

    contentAvailable: true,

    group: 'group',

    icon: 'icon',

    notificationCount: 3,

    notificationTimestamp: '2022-02-02T15:17:42.767Z',

    sound: 'chime.mp3',

    subtitle: 'subtitle1',

    tag: 'tag',

    threadIdentifier: 'threadIdentifier',

    titleLocArgs: ['titleArg1', 'titleArg2'],

    titleLocKey: 'titleKey',

});
```

Required properties

- `title`: (string) – The notification's title.
- `body`: (string) – The notification's body.
- `trigger`: (string) – The trigger that raised the notification message. Must be one of: push, location, calendar, timeInterval, other.

Optional properties

- `action`: (string) – The action associated with the notification.
- `attachments`: (array of `MessageNotificationAttachmentProps`) – Attachments added to the notification (they can be part of the data object).
- `bodyLocArgs` (array of string) - Variable string values to be used in place of the format specifiers in bodyLocArgs to use to localize the body text to the user's current localization.
- `bodyLocKey`: (string) – The key to the body string in the app's string resources to use to localize the body text to the user's current localization.
- `category`: (string) – The category associated to the notification.
- `contentAvailable`: (boolean) – The application is notified of the delivery of the notification if it's in the foreground or background, the app will be woken up (iOS only).
- `group`: (string) – The group which this notification is part of.
- `icon`: (string) – The icon associated to the notification (Android only).
- `notificationCount`: (integer) – The number of items this notification represent.
- `notificationTimestamp`: (string) – The time when the event of the notification occurred.
- `sound`: (string) – The sound played when the device receives the notification.
- `subtitle`: (string) – The notification's subtitle. (iOS only)
- `tag`: (string) – An identifier similar to 'group' but usable for different purposes (Android only).
- `threadIdentifier`: (string) – An identifier similar to 'group' but usable for different purposes (iOS only).
- `titleLocArgs`: (array of string) – Variable string values to be used in place of the format specifiers in titleLocArgs to use to localize the title text to the user's current localization.
- `titleLocKey`: (string) – The key to the title string in the app's string resources to use to localize the title text to the user's current localization.

#### MessageNotificationAttachmentProps

Attachment object that identifies an attachment in the Message Notification event.

**Required properties**

- `identifier`: (string) – Attachment identifier.
- `type`: (string) – Attachment type.
- `url`: (string) – Attachment URL.

For example:

```typescript
const attachment: MessageNotificationAttachmentProps = {

    identifier: 'id',

    type: 'type',

    url: 'http://att.url',

};
```

### Tracking structured events

Our philosophy in creating Snowplow is that users should capture important consumer interactions and design suitable data structures for this data capture. You can read more about that philosophy [here](/docs/event-studio/). Using `trackSelfDescribingEvent` captures these interactions with custom schemas, as desribed above.

However, as part of a Snowplow implementation there may be interactons where custom Self Describing events are perhaps too complex or unwarranted. They are then candidates to track using `trackStructuredEvent`, if none of the other event-specific methods outlined below are appropriate.

For example:

```typescript
tracker.trackStructuredEvent({

  category: 'my-category',

  action: 'my-action',

  label: 'my-label',

  property: 'my-property',

  value: 50.00

});
```

**Required properties**

- `category`: The name you supply for the group of objects you want to track e.g. ‘media’, ‘ecomm’
- `action`: A string which defines the type of user interaction for the web object e.g. ‘play-video’, ‘add-to-basket’

**Optional properties**

- `label`: (string) - identifies the specific object being actioned e.g. ID of the video being played, or the SKU or the product added-to-basket
- `property`: (string) - describes the object or the action performed on it. This might be the quantity of an item added to basket
- `value`: (number) - quantifies or further describes the user action. This might be the price of an item added-to-basket, or the starting time of the video where play was just pressed

---

# Track installs with the React Native tracker
> Track app installation events automatically on first launch with the React Native tracker installation autotracking feature.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/installation-tracking/

Installation tracking tracks an install event which occurs the first time an application is opened. The tracker will record when it's first been installed, so deleting and reinstalling an app will trigger another install event.

If installation autotracking is not enabled, the tracker will still keep track of when the app was first installed, so that when enabled, the tracker will send the recorded install event with a timestamp reflecting when it was first installed.

The installation autotracking is disabled by default. It can be set in `TrackerConfiguration` like in the example below:

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    installAutotracking: true, // disabled by default

});
```

---

# Track application lifecycle changes with the React Native tracker
> Automatically track foreground and background transitions with lifecycle events and attach lifecycle context entities to all events.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/

The tracker can capture application lifecycle state changes. In particular, when the app changes state from foreground to background and vice versa.

The lifecycle tracking is enabled by default. It can be disabled using the `lifecycleAutotracking` option like in the example below:

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    lifecycleAutotracking: true, // enabled by default

});
```

Once enabled, the tracker will automatically track a [`Background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event) when the app is moved to background and a [`Foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event) when the app moves back to foreground (becomes visible in the screen).

The tracker attaches a [`LifecycleEntity`](/docs/events/ootb-data/mobile-lifecycle-events/#lifecycle-entity) to all the events tracked by the tracker reporting if the app was visible (foreground state) when the event was tracked.

The `LifecycleEntity` value is conditioned by the internal state of the tracker only. To make an example, if the app is in foreground state but the developer tracks a `Background` event intentionally, it would force the generation of a `LifecycleEntity` that mark the app as non visible, even if it's actually visible in the device.

---

# Track platform and application data with the React Native tracker
> Track platform context with device information and application context with app version and build using the React Native tracker.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/

Platform and application data tracking features capture information about the device and the app.

They can be configured through `TrackerConfiguration` like in the example below:

```typescript
const tracker = await newTracker({

  namespace: 'appTracker',

  endpoint: COLLECTOR_URL,



  // application information

  appId: 'my-app-id',

  appVersion: '1.0.0',

  appBuild: '1',



  platformContext: true, // enabled by default

});
```

## Application information

### App ID

Set the application ID using the `appId` field of the tracker configuration object. This will be attached to every event the tracker fires. The information will be available in the `app_id` column of the events table.

### App version and build number

To track the application version, you can pass it using the `appVersion` configuration option. The version of can be a semver-like structure (e.g 1.1.0) or a Git commit SHA hash.

Additionally, you can track the build name of the application (e.g., `s9f2k2d` or `1.1.0 beta`) using the `build` option.

Depending on whether `build` property is configured, the information will be tracked in a context entity using either of these schemas:

1. If only `appVersion` is configured, [the Web `application` entity is used](/docs/events/ootb-data/app-information/#entity-definitions).
2. If both `appVersion` and `build` are configured, [the mobile `application` entity is used](/docs/events/ootb-data/app-information/#entity-definitions).

## Platform context

The [platform context entity](/docs/events/ootb-data/device-and-browser/#mobile-entity) contains information about the user's device.

By default only the following properties are tracked automatically:

| Property             | Type   | Description                                                                                                                                             | Required in schema |
| -------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `osType`             | String | Type of the operating system (e.g., "ios", "tvos", "watchos", "osx", "android")                                                                         | Yes                |
| `osVersion`          | String | Version of the mobile operating system.                                                                                                                 | Yes                |
| `deviceManufacturer` | String | Device vendor.                                                                                                                                          | Yes                |
| `deviceModel`        | String | Model of the device.                                                                                                                                    | Yes                |
| `language`           | String | System language currently used on the device (ISO 639)                                                                                                  | No                 |
| `resolution`         | String | Screen resolution in pixels. Arrives in the form of WIDTHxHEIGHT (e.g., 1200x900). Doesn't change when device orientation changes                       | No                 |
| `scale`              | Number | Scale factor used to convert logical coordinates to device coordinates of the screen (uses UIScreen.scale on iOS and DisplayMetrics.density on Android) | No                 |

The platform context entity can be used to track more information. However, you will need to pass it manually using callback functions.

```ts
const tracker = await newTracker({

  namespace: "sp1",

  endpoint: "http://example.com",

  // provide custom getters for the platform context properties

  platformContextRetriever: {

    getAppleIdfv: async () => "apple-idfv",

    getAppSetId: async () => "app-set-id",

    getAvailableStorage: async () => 1000,

  }

});
```

The following table lists all the extra properties that can be set:

| Property                | Type                                 | Description                                                                                                                                                             | Required in schema |
| ----------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `carrier`               | String                               | Carrier of the SIM inserted in the device.                                                                                                                              | No                 |
| `networkType`           | String                               | One of: "mobile", "wifi", "offline"                                                                                                                                     | No                 |
| `networkTechnology`     | String                               | Radio access technology that the device is using.                                                                                                                       | No                 |
| `openIdfa`              | String                               | Deprecated property.                                                                                                                                                    | No                 |
| `appleIdfa`             | String                               | Advertising identifier on iOS.                                                                                                                                          | No                 |
| `appleIdfv`             | String                               | UUID [identifier for vendors](https://developer.apple.com/documentation/uikit/uidevice/1620059-identifierforvendor) on iOS.                                             | No                 |
| `androidIdfa`           | String                               | Advertising identifier on Android.                                                                                                                                      | No                 |
| `physicalMemory`        | Integer                              | Total physical system memory in bytes                                                                                                                                   | No                 |
| `systemAvailableMemory` | Integer                              | Available memory on the system in bytes (Android only)                                                                                                                  | No                 |
| `appAvailableMemory`    | Integer                              | Amount of memory in bytes available to the current app (iOS only)                                                                                                       | No                 |
| `batteryLevel`          | Integer                              | Remaining battery level as an integer percentage of total battery capacity                                                                                              | No                 |
| `batteryState`          | String                               | Battery state for the device. One of: "unplugged", "charging", "full".                                                                                                  | No                 |
| `lowPowerMode`          | Boolean                              | A Boolean indicating whether Low Power Mode is enabled (iOS only)                                                                                                       | No                 |
| `availableStorage`      | Integer                              | Bytes of storage remaining                                                                                                                                              | No                 |
| `totalStorage`          | Integer                              | Total size of storage in bytes                                                                                                                                          | No                 |
| `isPortrait`            | Boolean                              | A Boolean indicating whether the device orientation is portrait (either upright or upside down)                                                                         | No                 |
| `appSetId`              | String                               | Android vendor ID scoped to the set of apps published under the same Google Play developer account (see <https://developer.android.com/training/articles/app-set-id>)   | No                 |
| `appSetIdScope`         | String (either "app" or "developer") | Scope of the `appSetId`. Can be scoped to the app or to a developer account on an app store (all apps from the same developer on the same device will have the same ID) | No                 |

---

# Track screen views with the React Native tracker
> Track screen views automatically with React Navigation or React Native Navigation, enable screen context entities, and measure screen engagement.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/screen-tracking/

Screen view tracking captures screen changes within the app.

To track a ScreenViewEvent, use the `trackScreenViewEvent` tracker method [as explained here](/docs/sources/react-native-tracker/tracking-events/#tracking-screen-view-events). For example:

```typescript
tracker.trackScreenViewEvent({

    name: 'my-screen-name',

    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',

    type: 'carousel',

    transitionType: 'basic'

});
```

## Tracking screen views in React Navigation

If you are using the [React Navigation](https://reactnavigation.org/) library for navigation in your app, you can implement automatic screen view tracking by adding a callback to your `NavigationContainer`. The steps are explained [in the documentation for React Navigation](https://reactnavigation.org/docs/screen-tracking/).

## Tracking screen views in React Native Navigation

When using the [React Native Navigation](https://wix.github.io/react-native-navigation/docs/before-you-start/) library for navigation in your app, you can automatically track screen views by registering a listener when your component appears on screen. Use the `Navigation.events().registerComponentDidAppearListener` callback to subscribe the listener and track screen views as [documented here](https://wix.github.io/react-native-navigation/api/events/#componentdidappear).

## Screen context entity

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    screenContext: true, // enabled by default

});
```

If the `screenContext` property is enabled, the tracker attaches a [`Screen` entity](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/screen/jsonschema/1-0-0) to all the events tracked by the tracker reporting the last (and probably current) screen visible on device when the event was tracked.

The `Screen` entity is conditioned by the internal state of the tracker only. To make an example, if the developer manually tracks a `ScreenView` event, all the following events will have a `Screen` entity attached reporting the same information as the last tracked ScreenView event.

## Screen engagement tracking

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    screenEngagementAutotracking: true, // enabled by default

});
```

Screen engagement tracking is a feature that enables tracking the user activity on the screen. This consists of the time spent and the amount of content viewed on the screen.

Concretely, it consists of the following metrics:

1. Time spent on screen while the app was in foreground (tracked automatically).
2. Time spent on screen while the app was in background (tracked automatically).
3. Number of list items scrolled out of all list items (requires some manual tracking).
4. Scroll depth in pixels (requires some manual tracking).

This information is attached using a [`screen_summary` context entity](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) to the following events:

1. [`screen_end` event](/docs/events/ootb-data/page-activity-tracking/#screen-end-event) that is automatically tracked before a new screen view event.
2. [`application_background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event).
3. [`application_foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event).

Screen engagement tracking is enabled by default, but can be configured using the `TrackerConfiguration.screenEngagementAutotracking` option.

For a demo of how mobile screen engagement tracking works in action, **[please visit this demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/)**.

#### Updating list item view and scroll depth information

To update the list item viewed and scroll depth information tracked in the screen summary entity, you can track the `ListItemView` and `ScrollChanged` events with this information. When tracked, the tracker won't send these events individually to the collector, but will process the information into the next `screen_summary` entity and discard the events. You may want to track the events every time a new list item is viewed on the screen, or whenever the scroll position changes.

To update the list items viewed information:

```js
tracker.trackListItemViewEvent({

    index: 1,

    itemsCount: 10,

});
```

To update the scroll depth information:

```js
tracker.trackScrollChangedEvent({

    yOffset: 10,

    xOffset: 20,

    viewHeight: 100,

    viewWidth: 200,

    contentHeight: 300,

    contentWidth: 400,

});
```

---

# Track sessions with the React Native tracker
> Track user sessions with client_session entities, configure foreground and background timeouts, and retrieve session data at runtime.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/session-tracking/

Session tracking captures the session which helps to keep track of the user activity in the app.

Client session tracking is enabled by default. It can be set using the `sessionContext` option as explained below.

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    sessionContext: true,

});
```

When enabled, the tracker appends a [`client_session` entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) to each event it sends and it maintains this session information as long as the application is installed on the device.

Sessions correspond to tracked user activity. A session expires when no tracking events have occurred for the amount of time defined in a timeout (by default 30 minutes). The session timeout check is executed for each event tracked. If the gap between two consecutive events is longer than the timeout the session is renewed. There are two timeouts since a session can timeout in the foreground (while the app is visible) or in the background (when the app has been suspended, but not closed).

The timeouts for the session can be configured like in the example below:

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    foregroundSessionTimeout: 1800, // seconds

    backgroundSessionTimeout: 1800 // seconds

});
```

The lifecycle events (`Foreground` and `Background` events) have a role in the session expiration. The lifecycle events can be enabled as explained in [App Lifecycle Tracking](/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/). Once enabled they will be fired automatically when the app moves from foreground state to background state and vice versa.

When the app moves from foreground to background, the `Background` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the foreground timeout. When the app moves from background to foreground, the `Foreground` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the background timeout.

For instance, with this configuration:

```typescript
const tracker = await newTracker({

    namespace: 'appTracker',

    endpoint: COLLECTOR_URL,

    foregroundSessionTimeout: 360, // seconds

    backgroundSessionTimeout: 15 // seconds

});
```

the session would expire if the app is in background for more than 15 seconds, like in this example:

```text
time: 0s - ScreenView event - foreground timeout session check - session 1

time: 3s - Background event - foreground timeout session check (3 < 360) - session 1

time: 30s - Foreground event - background timeout session check (30 > 15) - session 2
```

In the above example, the `Foreground` event triggers a new session because the time spent in background (without tracked events) is bigger than the background timeout for the session.

## Getting session data from the tracker

The React Native tracker has implemented callbacks that enable you to get session data back from the tracker at runtime. The way to do so is by using a tracker's `get..` methods. As with all tracker methods, these callbacks are also asynchronous, and they return a Promise that resolves to the respective value (or `undefined`) when fulfilled.

The available methods, are:

### getSessionUserId

This method returns a promise that resolves to the identifier (string UUIDv4) for the user of the session.

```javascript
const sessionUserId = await tracker.getSessionUserId();
```

### getSessionId

This method returns a promise that resolves to the identifier (string UUIDv4) for the session.

```javascript
const sessionId = await tracker.getSessionId();
```

### getSessionIndex

This method returns a promise to resolve to the index (number) of the current session for this user.

```javascript
const sessionIdx = await tracker.getSessionIndex();
```

### getIsInBackground

This method returns a promise to resolve to whether (boolean) the app is currently in background.

```javascript
const isInBackground = await tracker.getIsInBackground();
```

### getBackgroundIndex

This method returns a promise to resolve to the number of background transitions in the current session.

```javascript
const bgIndex = await tracker.getBackgroundIndex();
```

### getForegroundIndex

This method returns a promise to resolve to the number of foreground transitions in the current session.

```javascript
const fgIndex = await tracker.getForegroundIndex();
```

### getSessionState

This method returns a promise to resolve to the object containing the full state used to build the `client_session` entity in a single call, rather than the individual fields returned by the other methods.

```javascript
const sessionState = await tracker.getSessionState();
```

---

# Add entities and other data to events with the Roku tracker
> Add event context entities, subject data with user and platform information, and custom timestamps to Roku tracker events.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/adding-data/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset. Each of them involves a different class from the Roku tracker.

1. Event context using self-describing data: Attach event context, describing anything you like, in the form of self-describing JSONs.
2. Subject: Include information about the user, or the platform on which the event occurred.
3. Timestamp: Override the default event timestamp with your own timestamp.

You can attach any of these as additional arguments to the track event methods.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event’s entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`context`) in your data storage. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

```brightscript
m.global.snowplow.screenView = {

    id: "screen23",

    context: [

        {

            schema: "iglu:com.my_company/movie_poster/jsonschema/1-0-0",

            data: {

                movie_name: "Solaris",

                poster_country: "JP",

                poster_date: "1978-01-01"

            }

        },

        {

            schema: "iglu:com.my_company/customer/jsonschema/1-0-0",

            data: {

                p_buy: 0.23,

                segment: "young_adult"

            }

        }

    ]

}
```

## Adding user and platform data with Subject

Subject information describes the user and device associated with the event, such as their user ID, what type of device they used, or what size screen that device had.

This information can be entered during tracker initialization by passing a `subject` associative array to the `init` method. All of the information is optional.

Some subject information is filled automatically by the tracker. This includes the platform of the user, timezone, language, resolution, and viewport.

The following table lists all the properties that can be set in tracker initialization. These are all part of the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/).

If not provided, from v0.3.0 onwards the `domainUserId` and `domainSessionId` will be generated by the tracker and stored in the app [Registry](https://developer.roku.com/en-au/docs/developer-program/getting-started/architecture/file-system.md). This can be disabled by setting the `sessionContext` option to `false`. By default, this information is attached via the `client_session` entity, similar to mobile platforms. You can also include these values in the `domain_userid`, `domain_sessionid`, and `domain_sessionidx` fields by enabling the `sessionAsCookies` setting.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

| Property             | Description                                  |
| -------------------- | -------------------------------------------- |
| `userId`             | Unique user identifier in your system        |
| `domainUserId`       | Cookie-based unique identifier for user      |
| `domainSessionId`    | Cookie-based unique identifier for session   |
| `networkUserId`      | Cookie-based unique identifier for user      |
| `appId`              | Unique identifier for application            |
| `applicationContext` | Enable an entity with the channel version    |
| `sessionContext`     | Whether to enable or disable session state   |
| `sessionAsCookies`   | Enable session identifiers in atomic fields  |
| `userAnonymous`      | Use a nil ID for the `domainUserId` value    |
| `sessionAnonymous`   | Use a nil ID for the `domainSessionId` value |

Example:

```brightscript
m.global.snowplow.init = {

    subject: {

        userId: "...",

        domainUserId: "...",

        domainSessionId: "...",

        networkUserId: "...",

        appId: "...",

        applicationContext: true,

        sessionContext: false,

        sessionAsCookies: true,

        userAnonymous: true,

        sessionAnonymous: true,

    }

}
```

## Event timestamps

Processed Snowplow events have five different timestamps.

| Timestamp name     | Description                                                                           |
| ------------------ | ------------------------------------------------------------------------------------- |
| `dtm`              | Device timestamp. Added automatically during event creation.                          |
| `ttm`              | True timestamp. This can be manually set as an alternative to `dtm`.                  |
| `stm`              | Sent timestamp. Added automatically on event sending.                                 |
| `collector_tstamp` | Added by the event collector.                                                         |
| `etl_tstamp`       | Added after event enrichment during the processing pipeline.                          |
| `derived_tstamp`   | Either a calculated value (`collector_tstamp` - (`stm` - `dtm`)) or the same as `ttm` |

Overriding the default event timestamp (`dtm`) with `ttm` can be useful in some situations. For example, if the Snowplow event refers to an action that happened previously but is only now being tracked. The type of the timestamp is a string with number of milliseconds since the Unix epoch.

Example:

```brightscript
m.global.snowplow.structured = {

    se_ca: "category",

    se_ac: "action",

    se_va: 10,

    ttm: CreateObject("roDateTime").AsSeconds().ToStr() + "000"

}
```

---

# Configure how events are sent for the Roku tracker
> Configure Roku tracker network settings with collector endpoint, HTTP method, server anonymization, and tracker namespaces for multiple collectors.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/configuration/

When you initialize your tracker (using the `init` method), you will need to provide its network configuration as an associative array under the `network` property. The configuration consists of the collector endpoint and HTTP method type (see below table).

Example:

```brightscript
m.global.snowplow.init = {

    network: {

        collector: "http://...",

        method: "POST"

    }

}
```

Properties of the network configuration:

| Network configuration parameter | Description                                                 |
| ------------------------------- | ----------------------------------------------------------- |
| `collector`                     | URI for the Snowplow collector endpoint.                    |
| `method`                        | HTTP method to use. `GET` and `POST` methods are supported. |
| `serverAnonymous`               | Enable the `SP-Anonymous` header for server anonymization.  |

## Tracker namespaces

You may initialize multiple trackers, each with a different namespace. In this way, you can send events to multiple Snowplow collectors from your application. An example scenario for this would be sending events to both a staging and production pipeline. Each tracker processes events in its own background thread.

To initialize a tracker with a custom namespace, set the `namespace` property in the associative array passed to the `init` method. When no namespace is given, a default namespace is assigned to the tracker. Reinitializing trackers with the same namespace results in updating the configuration for the already initialized trackers. The following example initializes a tracker with the namespace "ns1":

```brightscript
m.global.snowplow.init = {

    namespace: "ns1",

    network: { ... }

}
```

Trackers can be individually addressed using their namespaces when tracking events. To send events to a specific tracker, call its namespace as follows: `m.global.snowplow.trackerNamespace.structured = {...}`. Here is an example of tracking a screen view event using a tracker with the namespace "ns1":

```brightscript
m.global.snowplow.ns1.screenView = {

    id: "screen23",

    name: "HUD > Save Game"

}
```

To track events with all initialized trackers, simply call methods to track events on the Snowplow instance without specifying a tracker namespace. The following example tracks a screen view event using all initialized trackers:

```brightscript
m.global.snowplow.screenView = {

    id: "screen23",

    name: "HUD > Save Game"

}
```

## Logging

The package makes use of [roku-log](https://github.com/georgejecook/roku-log), a logging framework for Roku Scenegraph apps. roku-log enables configuring the logging output and setting log levels for the severity of the output. Under the `debug` log level, the tracker outputs information about each request. On the other hand, setting the log level to `error` will only output crashes and tracking errors. Please refer to the roku-log [instructions](https://github.com/georgejecook/roku-log) to learn about setting it up.

---

# Roku device info context
> Automatically attach Roku device information including model, OS version, RIDA, network status, and display properties to all tracked events.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/device-context/

Device Info Context is an entity attached to events that provides information about the Roku device. It gives information about the device model and OS, adds unique identifiers for the channel and device, includes playback settings, current device usage, locale, display properties, network status, and device features.

The context is automatically added to all tracked events by default. If you prefer not to add the context to events, you may disable it using the `subject.deviceContext` option during tracker initialization:

```brightscript
m.global.snowplow.init = {

   subject: {

      deviceContext: false ' disabling device info context

   },

   ...

}
```

The context entity reflects the [ifDeviceInfo interface](https://developer.roku.com/en-gb/docs/references/brightscript/interfaces/ifdeviceinfo.md) in Roku SDK and picks some useful properties from it. It adds the following information:

| Attribute               | Type      | Description                                                                                 | Required? |
| ----------------------- | --------- | ------------------------------------------------------------------------------------------- | --------- |
| `model`                 | String    | Model name of the Roku device (e.g., 3940EU)                                                | yes       |
| `modelDisplayName`      | String    | Model display name (e.g., Roku Express 4K)                                                  | yes       |
| `modelType`             | String    | Type of device (STB or TV)                                                                  | yes       |
| `osVersion`             | String    | Roku OS version (e.g., 10.5.1.4059) (Roku OS ≥ 9.2)                                         | no        |
| `channelClientId`       | String    | A unique device identifier that is different across channels (Roku OS ≥ 8.1)                | no        |
| `isRIDADisabled`        | Boolean   | Indicates whether tracking via Roku's ID for Advertisers (RIDA) is disabled (Roku OS ≥ 8.1) | no        |
| `RIDA`                  | String    | A persistent unique identifier (UUID) for the device (Roku OS ≥ 8.1)                        | no        |
| `captionsMode`          | String    | Global captions are turned on or off, or are in instant replay mode                         | yes       |
| `audioOutputChannel`    | String    | Type of audio output                                                                        | yes       |
| `memoryLevel`           | String    | General memory level of the channel (normal, low, critical) (Roku OS ≥ 8.1)                 | no        |
| `timeSinceLastKeypress` | Integer   | Number of seconds since the last remote keypress was received                               | yes       |
| `userCountryCode`       | String    | ISO 3166-1 country code of the user's Roku account (Roku OS ≥ 8.1)                          | no        |
| `countryCode`           | String    | Roku Channel Store associated with a user's Roku account                                    | yes       |
| `videoMode`             | String    | Video playback resolution                                                                   | yes       |
| `displayWidth`          | Integer   | Physical width of the attached display in centimeters                                       | yes       |
| `displayHeight`         | Integer   | Physical height of the attached display in centimeters                                      | yes       |
| `displayProperties`     | String\[] | List of keys for display properties of the screen                                           | yes       |
| `connectionType`        | String    | Type of internet connection the device is using (e.g., WiFiConnection)                      | yes       |
| `internetStatus`        | Boolean   | Internet connection status of the device (Roku OS ≥ 10.0) .                                 | no        |
| `features`              | String\[] | List of features that the current device/firmware supports                                  | yes       |

The `displayProperties` list may include keys for the following display properties: `Internal` (display part of Roku player), `Hdr10` (display supports HDR10), `Hdr10Plus` (display supports HDR10), `DolbyVision` (display supports Dolby Vision). The `features` list of activated features of the device may include `5.1_surround_sound`, `can_output_5.1_surround_sound`, `sd_only_hardware`, `usb_hardware`, `sdcard_hardware`, `ethernet_hardware`, `gaming_hardware`, `energy_star_compliant`, `soundbar_hardware`, and `handsfree_voice`.

## Roku's ID for Advertisers (RIDA)

`RIDA` is a Universally Unique Identifier (UUID) that is unique and persistent for the Roku device. It is designed to generally follow the guidelines established for the IDFA (Identifier for Advertising) that are available on other platforms such as iOS and Android. The ID is intended for Roku publishers to enable frequency capping and targeted advertising on the Roku platform.

Users can disable RIDA tracking by selecting "Limit ad tracking" in the device settings. In this case, the `IsRIDADisabled` property will be true and `RIDA` will be a temporary ID that expires after 30 days.

RIDA is tracked by the tracker by default. However, there may be cases where you may not want to track the identifier. To disable tracking RIDA, deactivate the `subject.RIDATracking` option during tracker initialization:

```brightscript
m.global.snowplow.init = {

   subject: {

      RIDATracking: false ' disabling tracking RIDA

   },

   ...

}
```

---

# Example Roku app with Snowplow tracking
> Demo Roku video player channel showing Snowplow tracking implementation in BrightScript with setup and deployment instructions.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/example-app/

The tracker comes with a demo app that shows it in use. It is a simple video player with a fixed collection of played videos and an interface to switch between them. The demo app may serve you to understand how to embed Snowplow tracking within a BrightScript channel.

The project in included in the [tracker repository](https://github.com/snowplow-incubator/snowplow-roku-tracker), in the `src-demo-app` subfolder. The following steps to build and deploy the channel assume that you have enabled developer mode on your Roku device and it is connected to your network.

1. Download the [Roku tracker project](https://github.com/snowplow-incubator/snowplow-roku-tracker).
2. Run `npm install` within the project.
3. Install ropm `npm i ropm -g`.
4. Install ropm packages `ropm install`.
5. Create `.env` file with environment variables in the root of this repository with the following content:\
   `ROKU_IP=192.168.100.129`\
   `ROKU_PASSWORD=XXXX`
6. Add configuration for Snowplow collector to `src-demo-app/manifest`:\
   `snowplow_collector=http://192.168.100.127:9090`\
   `snowplow_method=POST`
7. Start the demo app using `npm run demo-app`.

Alternatively, you may run the demo app from Visual Studio Code as the debug configuration is already prepared. Install the BrightScript extension to Visual Studio Code and choose "Run demo app" in the debug options.

Events will be sent to the Snowplow collector as you navigate through the app.

---

# Roku tracker SDK
> Track events in BrightScript and BrighterScript Roku applications with SceneGraph channel support using the Snowplow Roku tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/

The Snowplow Roku Tracker allows you to track Snowplow events in your BrightScript and BrighterScript applications. The tracker is compatible with SceneGraph channels. The current version is 0.3.1.

The tracker is published on NPM as [@snowplow/roku-tracker](https://www.npmjs.com/package/@snowplow/roku-tracker) and can be installed with [ropm](https://github.com/rokucommunity/ropm). It may also be included in BrightScript apps directly by copying source files from releases [published on Github](https://github.com/snowplow-incubator/snowplow-roku-tracker).

We've created a simple [demo Roku channel](/docs/sources/roku-tracker/example-app/) to show you the tracker in use.

---

# Media tracking with the Roku tracker
> Enable automatic audio and video playback tracking for Roku Audio and Video nodes with support for media tracking v1 and v2 schemas.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/media-tracking/

Media tracking enables collecting events about audio/video playback from the [Audio](https://developer.roku.com/en-gb/docs/references/scenegraph/media-playback-nodes/audio.md) and [Video](https://developer.roku.com/en-gb/docs/references/scenegraph/media-playback-nodes/video.md) nodes provided in the Roku SceneGraph SDK for media playback. Once enabled for a node, the tracker subscribes to selected events and tracks them automatically. This makes adding media tracking into your Roku channels really simple.

There are [multiple versions](/docs/events/ootb-data/media-events/) of media tracking available, and the Roku tracker supports [v1](/docs/sources/roku-tracker/media-tracking/v1/) and (from v0.3.0) [v2](/docs/sources/roku-tracker/media-tracking/v2/).

---

# Roku media tracking (v1)
> Track Roku audio and video playback events with media tracking v1 schemas including player state changes and position tracking.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/media-tracking/v1/

Media tracking has [multiple versions](/docs/sources/web-trackers/tracking-events/media/) of schema available. This document is for version 1 of the schemas, it is recommended to [migrate to v2 tracking](/docs/sources/roku-tracker/media-tracking/v2/#migrating-from-media-v1-apis).

## Usage

To start media tracking for an Audio/Video node, assign a `roAssociativeArray` with the node to the `enableVideoTracking` property:

```brightscript
m.global.snowplow.enableVideoTracking = {

    video: m.Video

}
```

From tracker version 0.3.0:

- You can also use `enableAudioTracking` and pass the node via `audio` instead of `video`
- The v2 `enableMediaTracking` call and `media` option will also work, the tracker will use v1 tracking if the optional `version` option is defined as `1`

In addition to the node and version selector, the `enableVideoTracking` property accepts several optional attributes listed in the tables below.

| Attribute                   | Type                  | Description                                                                                                       | Required?                                        |
| --------------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| `media` / `audio` / `video` | Audio / Video         | Audio/Video node to be tracked                                                                                    | yes                                              |
| `version`                   | Integer               | Tracking schema version to use; defaults to 1 if not provided for `enableVideoTracking` and `enableAudioTracking` | no                                               |
| `label` / `options.label`   | String                | An identifiable custom label sent with the event                                                                  | no                                               |
| `options.captureEvents`     | String\[]             | Types of events to capture                                                                                        | no, defaults to all events except for `position` |
| `options.boundaries`        | Integer\[]            | Percentage boundaries in playback for which events will be sent                                                   | no, defaults to `[10, 25, 50, 75]`               |
| `options.positionInterval`  | Integer               | Interval in seconds in which `position` events should be reported                                                 | no, defaults to 5                                |
| `context` / `entities`      | SelfDescribingJSON\[] | Array of custom entities to include with all generated media events                                               | no                                               |

To stop tracking events from the node, set the `disableVideoTracking`/`disableAudioTracking` property. The tracker will then stop observing and tracking events from the node. The property should be set with an `roAssociativeArray` with exactly one attribute, `video` (or `audio`/`media`), like so:

```brightscript
m.global.snowplow.disableVideoTracking = {

    video: m.video

}
```

## Media Player Events

### Media Events

All playback events are tracked automatically and conform to the `media_player_event` schema. The schema has two properties:

| Property | Description               | Required? |
| -------- | ------------------------- | --------- |
| `type`   | Type of event             | Yes       |
| `label`  | Identifiable custom label | No        |

The events are enriched with two context entities. The first is the `media_player` context entity that provides the following properties:

| Property          | Description                                                                             | Required? |
| ----------------- | --------------------------------------------------------------------------------------- | --------- |
| `currentTime`     | The current playback time                                                               | Yes       |
| `duration`        | A double-precision floating-point value indicating the duration of the media in seconds | No        |
| `ended`           | If playback of the media has ended                                                      | Yes       |
| `isLive`          | If the media is live                                                                    | Yes       |
| `loop`            | If the video should restart after ending                                                | Yes       |
| `muted`           | If the media element is muted                                                           | Yes       |
| `paused`          | If the media element is paused                                                          | Yes       |
| `percentProgress` | The percent of the way through the media                                                | No        |
| `playbackRate`    | Playback rate (1 is normal)                                                             | No        |
| `volume` .        | Volume percent                                                                          | Yes       |

The second is the Roku `video` context with these properties:

| Property               | Description                                                                                                             | Required? |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------- | --------- |
| `videoId`              | ID generated when video tracking of the video node was initialized.                                                     | Yes       |
| `contentId`            | ID of video provided in content metadata.                                                                               | No        |
| `contentTitle`         | Title of video provided in content metadata.                                                                            | No        |
| `contentUrl`           | URL of video provided in content metadata.                                                                              | No        |
| `contentType`          | Category of video (e.g., movie, season, series) provided in content metadata.                                           | No        |
| `streamFormat`         | Container format of video (e.g., mp4, wma, mkv) provided in content metadata.                                           | No        |
| `streamUrl`            | URL of the current stream.                                                                                              | No        |
| `measuredBitrate`      | Measured bitrate (bps) of the network when the stream was selected.                                                     | No        |
| `streamBitrate`        | Current bitrate of the stream.                                                                                          | No        |
| `isUnderrun`           | Indicates whether the stream was downloaded due to an underrun.                                                         | No        |
| `isResumed`            | Indicates whether the playback was resumed after trickplay.                                                             | No        |
| `videoFormat`          | Video codec of the currently playing video stream (e.g., hevc, mpeg2, mpeg4\_15).                                       | No        |
| `timeToStartStreaming` | Time in seconds from playback being started until the video actually began playing.                                     | No        |
| `width`                | Width of the video play window in pixels. 0 if the play window is set to the width of the entire display screen.        | Yes       |
| `height`               | Height of the video play window in pixels. 0 if the play window is set to the height of the entire display screen.      | Yes       |
| `errorStr`             | A diagnostic message indicating a video play error. Refer to the Roku Video documentation for the format of the string. | No        |

## Captured Types of Events

The tracker automatically captures several types of events. You may configure which types of events to track by passing a list of the types into the `options.captureEvents` property during tracker initialization:

```brightscript
m.global.snowplow.enableVideoTracking = {

    video: m.Video,

    options: {

        captureEvents: ["playing", "paused"]

    }

}
```

Overall, the types of events can be grouped into two categories: playback position events, and state change events.

### Playback Position Events

These "ping" events report changes in the playback position of the video while playing. There are two types of the events:

1. `percentprogress` events report when the playback reaches certain boundaries defined as percentages of the total playback duration.
2. `position` events are regular events sent when the playback changes by a certain number of seconds.

#### Event Type: `percentprogress`

Percent progress events are sent when predefined percentage boundaries in playback are reached. The boundaries may be configured using a `boundaries` list when initializing video tracking:

```brightscript
m.global.snowplow.enableVideoTracking = {

    video: m.Video,

    options: {

        captureEvents: ["percentprogress"],

        boundaries: [25, 50, 75, 100]

    }

}
```

The tracker sends percent progress events by default for the following percentage boundaries: 10%, 25%, 50%, and 75% of total playback.

These events can not be reported for live streams since they assume that the total duration of the video is known.

#### Event Type: `position`

Position events report the current playback position in regular intervals. These events are not reported by default. They can be activated using the `options.captureEvents` property.

The events are sent whenever the playback head changes by more than the predefined position interval. The interval defaults to 5 seconds. It can be configured using the `options.positionInterval` property:

```brightscript
m.global.snowplow.enableVideoTracking = {

    video: m.Video,

    options: {

        captureEvents: ["position"],

        positionInterval: 15 ' seconds

    }

}
```

These events do not require the total duration of the video to be known and can be reported for live streams as well.

### State Change Events

State change events are sent when the state of the video node changes due to interaction from the user, changes in the playback, or through any other influence. Transitions between the following states are captured:

- `playing`
- `paused`
- `buffering`
- `stopped`
- `finished`
- `error`
  - Error occured in the playback. The `errorStr` property in the `video` context will provide more details on the error.

The event type of the events (`type` property) reflects the current state.

All of the state changes are reported by default. In order to capture only selected states, use the `options.captureEvents` property:

```brightscript
m.global.snowplow.enableVideoTracking = {

    video: m.Video,

    options: {

        captureEvents: ["playing", "paused"]

    }

}
```

---

# Roku media tracking (v2)
> Track Roku media playback with v2 schemas including session entities, ping events, and percentage progress tracking for audio and video nodes.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/media-tracking/v2/

> **Note:** Media tracking has [multiple versions](/docs/sources/web-trackers/tracking-events/media/) of schemas available. This document is for version 2 of the schemas; tracker versions earlier than v0.3.0 [only support v1 tracking](/docs/sources/roku-tracker/media-tracking/v1/).

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

```brightscript
' 1. Start tracking when player loads (events are auto-tracked)

m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  label: "My Video Title"

}



' 2-3. Player state and events are automatically tracked from the Video node



' 4. End tracking when done

m.global.snowplow.disableMediaTracking = {

  media: m.Video

}
```

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

The Roku tracker automatically reads most player properties from the Audio/Video node. Configure additional options in `enableMediaTracking`:

```brightscript
m.global.snowplow.enableMediaTracking = {

  media: m.Video,           ' Audio/Video node to track (required)

  version: 2,               ' Schema version (defaults to 2)

  id: "session-123",        ' Custom session ID (auto-generated if not set)

  label: "My Video",        ' Human-readable title for the media_player entity

  rokuVideoContext: true    ' Whether to include the roku_video_context entity

}
```

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

```brightscript
m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  pings: true,        ' Enabled by default

  pingInterval: 30    ' Seconds between pings (default: 30)

}



' Or turn off pings entirely

m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  pings: false

}
```

### Percentage progress

Track events when playback reaches specified percentage boundaries.

```brightscript
m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  boundaries: [25, 50, 75]  ' By default, the tracker will fire events at these percentages

}
```

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

```brightscript
m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  sessions: true

}
```

### Custom entities

You can attach custom entities to all events for this media tracking instance.

```brightscript
m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  entities: [

      {

          schema: "iglu:com.example/video_metadata/jsonschema/1-0-0",

          data: {"contentId": "abc123", "category": "tutorial"}

      }

  ]

}
```

### Filter events

Only track specific event types by providing an allowlist:

```brightscript
m.global.snowplow.enableMediaTracking = {

  media: m.Video,

  version: 2,

  captureEvents: ["play", "pause", "end"]

}
```

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

The Roku tracker automatically tracks the following events from Audio/Video nodes:

- Ready, play, pause, end events
- Seek start, seek end events
- Ping events
- Percent progress events (if configured)
- Buffer start, buffer end events
- Quality change events
- Error events

To track events manually or track events not auto-tracked, use `trackMediaEvent`:

```brightscript
m.global.snowplow.trackMediaEvent = {

  media: m.Video,

  schema: "iglu:com.snowplowanalytics.snowplow.media/ad_click_event/jsonschema/1-0-0",

  data: { "percentProgress": 50 }

}
```

### Player state events

```brightscript
m.global.snowplow.trackMediaEvent = {

  media: m.Video,

  schema: "iglu:com.snowplowanalytics.snowplow.media/playback_rate_change_event/jsonschema/1-0-0",

  data: { "newRate": 1.5 }

}



m.global.snowplow.trackMediaEvent = {

  media: m.Video,

  schema: "iglu:com.snowplowanalytics.snowplow.media/volume_change_event/jsonschema/1-0-0",

  data: { "newVolume": 80 }

}
```

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

```brightscript
' Ad break start

m.global.snowplow.trackMediaEvent = {

  media: m.Video,

  schema: "iglu:com.snowplowanalytics.snowplow.media/ad_break_start_event/jsonschema/1-0-0",

  data: {},

  adBreak: {

      "breakId": "break-1",

      "name": "pre-roll",

      "breakType": "linear",

      "podSize": 3

  }

}



' Ad start

m.global.snowplow.trackMediaEvent = {

  media: m.Video,

  schema: "iglu:com.snowplowanalytics.snowplow.media/ad_start_event/jsonschema/1-0-0",

  data: {},

  ad: {

      "adId": "ad-1",

      "name": "Product Ad",

      "duration": 30

  }

}
```

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

```brightscript
m.global.snowplow.trackMediaEvent = {

  media: m.Video,

  schema: "iglu:com.example/video_interaction/jsonschema/1-0-0",

  data: { "action": "share", "platform": "twitter" }

}
```

## Update player state

Manual player state updates aren't needed for the Roku tracker. The tracker automatically tracks player state from the registered media node.

## Roku video entity

By default, the Roku tracker attaches a `video` entity to all media events.

### `video`

**Type:** Schema

Schema for a Roku video event (reflects the Video node: https\://developer.roku.com/en-gb/docs/references/scenegraph/media-playback-nodes/video.md)

**Properties:**

| Property                        | Description                                                                                                                         |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `videoId` _string_              | _Required._ ID generated when video tracking of the video node was initialized.                                                     |
| `contentId` _string_            | _Optional._ ID of video provided in content metadata.                                                                               |
| `contentTitle` _string_         | _Optional._ Title of video provided in content metadata.                                                                            |
| `contentUrl` _string_           | _Optional._ URL of video provided in content metadata.                                                                              |
| `contentType` _string_          | _Optional._ Category of video (e.g., movie, season, series) provided in content metadata.                                           |
| `streamFormat` _string_         | _Optional._ Container format of video (e.g., mp4, wma, mkv) provided in content metadata.                                           |
| `streamUrl` _string_            | _Optional._ URL of the current stream.                                                                                              |
| `measuredBitrate` _integer_     | _Optional._ Measured bitrate (bps) of the network when the stream was selected.                                                     |
| `streamBitrate` _integer_       | _Optional._ Current bitrate of the stream.                                                                                          |
| `isUnderrun` _boolean_          | _Optional._ Indicates whether the stream was downloaded due to an underrun.                                                         |
| `isResumed` _boolean_           | _Optional._ Indicates whether the playback was resumed after trickplay.                                                             |
| `videoFormat` _string_          | _Optional._ Video codec of the currently playing video stream (e.g., hevc, mpeg2, mpeg4\_15).                                       |
| `timeToStartStreaming` _number_ | _Optional._ Time in seconds from playback being started until the video actually began playing.                                     |
| `width` _integer_               | _Required._ Width of the video play window in pixels. 0 if the play window is set to the width of the entire display screen.        |
| `height` _integer_              | _Required._ Height of the video play window in pixels. 0 if the play window is set to the height of the entire display screen.      |
| `errorStr` _string_             | _Optional._ A diagnostic message indicating a video play error. Refer to the Roku Video documentation for the format of the string. |

## Migrating from media v1 APIs

If you are migrating from the [v1 media tracking](/docs/sources/roku-tracker/media-tracking/v1/), v2 has compatibility options to make the change easier.

For backwards compatibility with the earlier tracking API, you can continue using `enableVideoTracking` or `enableAudioTracking` instead of `enableMediaTracking`, and pass the node via the `audio`/`video` option instead of `media`. For most other options, v2 will accept either v1 or v2 `roAssociativeArray` formats. The tracking will work with any combination of these method and option names regardless of node type, as long as `version: 2` is specified. You can also specifically opt into version 1 by passing `version: 1` for any of these methods, though v2 options may not be recognized.

If using specific `version` values, the same version should also be specified for the corresponding `disable*Tracking` method. Similarly, mixing `enableVideoTracking` and `disableMediaTracking` will not work unless versions are specified consistently.

The additional options differ by version; but version 2 will accept the options from version 1 if it can't find the newer equivalent settings.

| v2 Attribute                | v1 Fallback                 | Type                  | Description                                                                                   |
| --------------------------- | --------------------------- | --------------------- | --------------------------------------------------------------------------------------------- |
| `media` / `audio` / `video` | `media` / `audio` / `video` | Audio / Video         | Audio/Video node to be tracked                                                                |
| `version`                   | `version`                   | Integer               | Tracking schema version to use; should be set to 2 if not using `enableMediaTracking`         |
| `id`                        | N/A                         | UUID                  | A unique ID to use for the media `session` entity to group all tracked events                 |
| `label`                     | `options.label`             | String                | An identifiable custom label sent with the event in the `media_player` entity                 |
| `sessions`                  | Default enabled             | boolean               | Whether to attach the media `session` entity on tracked events                                |
| `pings`                     | Default enabled             | boolean               | Whether to periodically generate media `ping_event`s while content is playing                 |
| `pingInterval`              | `options.positionInterval`  | Integer               | Interval in seconds in which `ping_event`s should be reported                                 |
| `boundaries`                | `options.boundaries`        | Integer\[]            | Percentage boundaries in playback for which `percentage_progress_event`s will be sent         |
| `captureEvents`             | `options.captureEvents`     | String\[]             | Types of events to capture. If specified, the event names should not have the `_event` suffix |
| `context` / `entities`      | `context`                   | SelfDescribingJSON\[] | Array of custom entities to include with all generated events                                 |

---

# Getting started with the Roku tracker
> Install the Roku tracker using ropm or manual installation, initialize tracking, and track structured events in your BrightScript app.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/quick-start-guide/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

The following steps will guide you through setting up the Roku tracker in your project and tracking a simple event.

## Installation

There are two options to install the Roku tracker package to your project:

1. using Roku package manager (ropm),
2. by manually copying package files.

### Using Roku Package Manager (ropm)

[ropm](https://github.com/rokucommunity/ropm) is a package manager for the Roku platform. If you are using ropm in your project, you may install the Roku tracker using the following command:

```bash
ropm install snowplow@npm:@snowplow/roku-tracker
```

### Manual Installation

The Roku tracker may be installed by simply copying source files to your Roku project. You may download and unpack the `dist.zip` or `dist.tar.gz` package from the latest release build on Github. Copy the following folders and files to your Roku project:

1. Contents of `dist/source` into your `source` directory
2. Contents of `dist/components` into your `components` directory

## Implementation

It is recommended that you instantiate Snowplow and add it to the global scope. In this way, it will be accessible from anywhere within your SceneGraph application. You may create the instance in the `init` function of your main scene.

If you installed the package using ropm, mount the component as follows:

```brightscript
m.global.AddField("snowplow", "node", false)

m.global.snowplow = CreateObject("roSGNode", "snowplow_Snowplow")
```

If you installed the package manually, mount the component as follows:

```brightscript
m.global.AddField("snowplow", "node", false)

m.global.snowplow = CreateObject("roSGNode", "Snowplow")
```

## Initialization

Trackers are initialized by setting the `init` property with configuration of the tracker. This configuration takes the form of a `roAssociativeArray`. At its most basic, the configuration takes a Snowplow collector endpoint like so:

```brightscript
m.global.snowplow.init = {

    network: {

        collector: "http://..."

    }

}
```

To learn more about configuring how events are sent, check out [this page](/docs/sources/roku-tracker/configuration/).

## Tracking events

To track an event, simply assign its properties as a `roAssociativeArray` to a field corresponding to the event type. For instance, to track a structured event, assign the `structured` property:

```brightscript
m.global.snowplow.structured = {

    se_ca: "category",

    se_ac: "action",

    se_la: "label",

    se_pr: "property",

    se_va: 10

}
```

Visit documentation about [tracking events](/docs/sources/roku-tracker/tracking-events/) to learn about other supported event types. You may also want to read about [adding more data to tracked events](/docs/sources/roku-tracker/adding-data/).

## Testing

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

Check out the [example Roku channel](/docs/sources/roku-tracker/example-app/) to see the tracker used with Snowplow Micro.

---

# Tracking events with the Roku tracker
> Track self-describing events, structured events, and screen views with the Roku tracker SDK including automatic install event tracking.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/tracking-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in methods to help you track different kinds of events. The methods range from single purpose methods, such as `screenView`, to the more complex but flexible `selfDescribing`, which can be used to track any kind of user behavior. We strongly recommend using `selfDescribing` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

Tracking methods supported by the Roku Tracker:

| Method           | Event type tracked                                  |
| ---------------- | --------------------------------------------------- |
| `selfDescribing` | Custom event based on "self-describing" JSON schema |
| `structured`     | Semi-custom structured event                        |
| `screenView`     | View of screen                                      |

All the methods share common features and parameters. Every type of event can have an optional context added. A Timestamp can also be provided for all event types to override the default event timestamp. See the next page to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event tracking methods, followed by the out-of-the-box event types. Note that you can also design and create your own page view, or screen view, using `selfDescribing`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

## Track self-describing events with `selfDescribing`

Use `selfDescribing` to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

The `selfDescribing` method takes a `roAssociativeArray`. This array takes a schema name and a flat hash of event data.

Example (assumes that you mounted the Snowplow instance in `m.global.snowplow`):

```brightscript
m.global.snowplow.selfDescribing = {

    data: {

        saveId: "4321",

        level: 23,

        difficultyLevel: "HARD",

        dlContent: true

    },

    schema: "iglu:com.example_company/save_game/jsonschema/1-0-2"

}
```

## Track structured events with `structured`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category" (`se_ca`), "action" (`se_ac`), "label" (`se_la`), and "value" (`se_va`) properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| Argument | Description                                                    | Required in event? |
| -------- | -------------------------------------------------------------- | ------------------ |
| `se_ca`  | The grouping of structured events which this action belongs to | Yes                |
| `se_ac`  | Defines the type of user interaction which this event involves | Yes                |
| `se_la`  | Often used to refer to the 'object' the action is performed on | No                 |
| `se_pr`  | Describing the 'object', or the action performed on it         | No                 |
| `se_va`  | Provides numerical data about the event                        | No                 |

Example:

```brightscript
m.global.snowplow.structured = {

    se_ca: "shop",

    se_ac: "add-to-basket",

    se_pr: "pcs",

    se_va: 2

}
```

## Track screen views with `screenView`

Use `screenView` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name`, `id`, `type`, and `transitionType`; while all are optional, you must provided at least one of either `name` or `id` to create a valid event. "Name" is the human-readable screen name, and "ID" should be the unique screen ID.

This method creates an unstruct event, by creating and tracking a self-describing event. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the parameters which you provide. That schema is hosted on the schema repository Iglu Central, and so will always be available to your pipeline.

Example:

```brightscript
m.global.snowplow.screenView = {

    id: "screen23",

    name: "HUD > Save Game"

}
```

## Automatic application install events

When [configuring the tracker](/docs/sources/roku-tracker/configuration/), use the `trackInstall` setting to opt into automatic [`application_install` tracking](/docs/events/ootb-data/mobile-lifecycle-events/#install-events).

There's no way to directly detect if the app is running as part of a fresh installation, so the tracker uses an indirect trigger. It will send an install event after it creates a first `userId` value for the [`client_session` entity](/docs/sources/roku-tracker/adding-data/#adding-user-and-platform-data-with-subject). Generating a `userId` occurs even if the `client_session` entity is not enabled. This is similar to the approach [recommended by Roku](https://community.roku.com/discussions/developer/tracking-application-install-and-version-state/188117).

```brightscript
m.global.snowplow.init = {

    trackInstall: true

}
```

> **Note:** This will track an "install" the first time the tracker runs and generates a user ID. If you are adding tracking to an existing channel or upgrading from an older SDK version, enabling this setting may create a burst of events as older installations retroactively report this event for the first time. To reduce this:
>
> 1. First release the app with Roku tracker v0.3+ and this setting **disabled**
> 2. Allow user devices time to generate their first `userId` values
> 3. Release an additional version with this setting **enabled**
> 4. Only new installations should track the event
>
> To avoid multiple releases, you could also only enable the setting conditionally based on a timestamp, so only installations after a certain date trigger the event.

---

# Add entities and other data to events with the Ruby tracker
> Add event context entities, subject data with user and platform information, page data, and custom timestamps to Ruby tracker events.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/adding-data-events/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset. Each of them involves a different class from the Ruby tracker.

1. [Event context](#event-context) using SelfDescribingJson. Attach event context, describing anything you like, in the form of self-describing JSONs.
2. [Subject](#adding-user-and-platform-data-with-subject): Include information about the user, or the platform on which the event occurred.
3. [Page](#adding-page-data-with-page): For websites, include data about the page on which the event occurred.
4. [Timestamp](#event-timestamps): Override the default event timestamp with your own timestamp.

You can attach any of these as additional arguments to the `track_x_event` methods.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same [self-describing JSON schemas](/docs/fundamentals/schemas/) as the [self-describing events](/docs/sources/ruby-tracker/tracking-events/). Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event's entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`contexts`) in your data storage. There is no limit to how many entities can be attached to one event.

The context entities were originally called "context" themselves, with the event context referred to as "contexts". This was confusing (and not great English grammar), so we changed the name. However, the name "contexts" persists in some places.

Check out our [demo Rails app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples) to see an example of a custom eCommerce event created using a self-describing event plus product entity. Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information - reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

**v0.7.0+:**

```ruby
# Tracking a screen view with context made up of two entities

entity1 = SnowplowTracker::SelfDescribingJson.new(

  'iglu:com.my_company/movie_poster/jsonschema/1-0-0',

  { 'movie_name' => 'Solaris',

    'poster_country' => 'JP',

    'poster_year$dt' => Date.new(1978, 1, 1) }

)

entity2 = SnowplowTracker::SelfDescribingJson.new(

  'iglu:com.my_company/customer/jsonschema/1-0-0',

  { 'p_buy' => 0.23,

    'segment' => 'young adult' }

)



tracker.track_screen_view(id: 'sci-123-abc', context: [entity1, entity2])
```

**Before v0.7.0:**

```ruby
# Tracking a screen view with context made up of two entities

entity1 = SnowplowTracker::SelfDescribingJson.new(

  'iglu:com.my_company/movie_poster/jsonschema/1-0-0',

  { 'movie_name' => 'Solaris',

    'poster_country' => 'JP',

    'poster_year$dt' => Date.new(1978, 1, 1) }

)

entity2 = SnowplowTracker::SelfDescribingJson.new(

  'iglu:com.my_company/customer/jsonschema/1-0-0',

  { 'p_buy' => 0.23,

    'segment' => 'young adult' }

)



tracker.track_screen_view(nil, 'sci-123-abc', [entity1, entity2])
```

***

> **Note:** The event context is always an array, even if you are only adding one entity.

## Adding user and platform data with Subject

Subject objects store information about the user associated with the event, such as their `user_id`, what type of device they used, or what size screen that device had. Also, they store which platform the event occurred on - e.g. server-side app, mobile, games console, etc.

Each Tracker is initialized with a Subject. This means that every event by default has the platform `srv`: server-side app. Platform is the only preset Subject parameter, which can be overriden using the `set_platform` method. All other parameters must be set manually. This can be done directly on the Subject, or, if it is associated with a Tracker, via the Tracker, which has access to all the methods of its Subject.

Your server-side code may not have access to all these parameters, or they might not be useful to you. All the Subject parameters are optional, except `platform`.

You could create and define a new Subject for every user or every event you want to track. Attaching that Subject to the `track_x_event` method call would add those properties to your event. Alternatively, you could swap the tracker-associated Subject for a more appropriate one before tracking the event, using `set_subject`.

The following table lists all the properties that can be set via Subject. These are all part of the [Snowplow Tracker Protocol](/docs/events/). Check out the [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Subject.html) for the details of how to set these parameters.

| Property                                   | **Description**                                                   |
| ------------------------------------------ | ----------------------------------------------------------------- |
| `platform`                                 | The platform the app runs on                                      |
| `user_id`                                  | Unique identifier for user                                        |
| `domain_userid`                            | Cookie-based unique identifier for user                           |
| `network_userid`                           | Cookie-based unique identifier for user                           |
| `domain_sessionid`                         | Cookie-based unique identifier for a visit/session of a `user_id` |
| `domain_sessionidx`                        | Cookie-based count of separate visits/sessions from a `user_id`   |
| `user_fingerprint`                         | User identifier based on (hopefully unique) browser features      |
| `user_ipaddress`                           | User's IP address                                                 |
| `useragent`                                | User agent or browser string                                      |
| `br_lang`                                  | The device/browser language                                       |
| `os_timezone`                              | The device OS's timezone                                          |
| `dvce_screenheight` and `dvce_screenwidth` | The device screen resolution                                      |
| `br_viewwidth` and `br_viewheight`         | The browser viewport size                                         |
| `br_colordepth`                            | The browser color depth                                           |

> **Note:** The methods for defining `domain_sessionid` and `domain_sessionidx` were added in tracker version 0.7.0.

Example:

**v0.7.0+:**

> **Note:** The ability to add `Subject` objects to `track_x_event` method calls (option 1 below) was added in version 0.7.0.

```ruby
# Creating a Subject and adding user data

subject = SnowplowTracker::Subject.new

subject.set_user_id('12345').set_timezone('Europe/London')



# Different ways to add Subject properties to an event

# 1. Providing the Subject as an argument

tracker.track_page_view(page_url: 'www.example.com', subject: subject)



# 2. Swapping the tracker-associated Subject for the new one

# This will affect all subsequent events tracked

tracker.set_subject(subject).track_page_view(page_url: 'www.example.com')



# 3. Setting the user data directly on the tracker-associated Subject

# This will affect all subsequent events tracked

tracker.set_user_id('12345').set_timezone('Europe/London')

tracker.track_page_view(page_url: 'www.example.com')
```

**Before v0.7.0:**

```ruby
# Creating a Subject and adding user data

subject = SnowplowTracker::Subject.new

subject.set_user_id('12345').set_timezone('Europe/London')



# Different ways to add Subject properties to an event

# Either way will affect all subsequent events tracked

# 1. Swapping the tracker-associated Subject for the new one

tracker.set_subject(subject).track_page_view('www.example.com')



# 2. Setting the user data directly on the tracker-associated Subject

tracker.set_user_id('12345').set_timezone('Europe/London')

tracker.track_page_view('www.example.com')
```

***

### Cookie-based user properties

Several of the user properties listed in the above table are relevant only to web apps, and are intended to derive from cookies. The Snowplow cookies are explained on [this page](/docs/sources/web-trackers/cookies-and-local-storage/). The client-side Snowplow JavaScript tracker sets first-party cookies, and the event collector sets a third-party cookie. As a server-side tracker, the Ruby tracker doesn't automatically have access to cookies. However, these properties can be extremely useful, e.g. in stitching events together from individual users, or for matching server-side events with client-side events.

Read more about sharing data between client-side and server-side trackers in this [blog post](https://snowplowanalytics.com/blog/2021/11/09/the-unrivaled-power-of-joining-client-and-server-side-tracking/).

If you have implemented the Ruby tracker as well as the JavaScript tracker in your web app, you could extract the values from the cookies, and set them in your Ruby events using these Subject methods. An example of this is included for `domain_user_id` in our [demo Rails app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples).

Extracting the `domain_user_id` from the `_sp_id` cookie in Rails:

```ruby
def snowplow_domain_userid

  sp_cookie = cookies.find { |key, _value| key =~ /^_sp_id/ }

  sp_cookie.last.split(".").first if sp_cookie.present?

end
```

Similarly, the `domain_session_id` and `domain_session_idx` values are saved in the first-party `_sp_ses` and `_sp_id` cookies. If you configured the JavaScript tracker with a different “cookieName” option, then these cookies will be named differently.

The `network_user_id` derives from the event collector's third-party cookie, hence the name "network" as it is set at a network level. It is the server-side user identifier. The cookie is named `sp` (Snowplow Micro pipelines call it `micro` instead). The default behavior is for the collector to provide a new cookie/network user ID for each event it receives. You can override the collector cookie's value with your own generated ID using the `set_network_user_id` method.

The user fingerprint is also a client-side concept. The JavaScript Tracker generates a fingerprint based on browser features and attaches it to all client-side events. You could develop your own fingerprints to attach to your Ruby server-side events with `set_fingerprint`.

## Adding page data with Page

If the Ruby tracker is incorporated into a website server, the events tracked will describe user activity on specific webpages. Knowing on which page an event occurred can be very valuable.

Add page URL, page title and referrer URL to any event by adding a Page object to any `track_x_event` method call. These parameters, of course, form the basis of the page view event. By directly adding page details to your other event types, you can avoid needing to JOIN onto your page view events to find out where they occurred.

Example:

```ruby
# Adding Page data to a struct event

page = SnowplowTracker::Page.new(page_url: 'http://www.example.com/register')



tracker.track_struct_event(category: 'forms',

                           action: 'start-input',

                           page: page)
```

The Page class was added in tracker version 0.7.0.

## Event timestamps

Processed Snowplow events have five different timestamps. They can have either `dvce_created_tstamp` or `true_tstamp`.

| **Timestamp name**    | **Description**                                                                                                        |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `dvce_created_tstamp` | Added during event creation                                                                                            |
| `true_tstamp`         | This can be manually set as an alternative to `dvce_created_tstamp`                                                    |
| `dvce_sent_tstamp`    | Added by the Emitter on event sending                                                                                  |
| `collector_tstamp`    | Added by the event collector                                                                                           |
| `etl_tstamp`          | Added after event enrichment during the processing pipeline                                                            |
| `derived_tstamp`      | Either a calculated value (`collector_tstamp - (dvce_sent_tstamp - dvce_created_tstamp)`) or the same as `true_tstamp` |

Overriding the default event timestamp (`dvce_created_tstamp`) can be useful in some situations. For example, if the Snowplow event refers to an action that happened previously but is only now being tracked. This can be achieved using the Ruby tracker classes DeviceTimestamp and TrueTimestamp (see [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Timestamp.html)).

---

# Configure how events are sent for the Ruby tracker
> Configure Ruby emitters and async emitters with collector endpoint, HTTP method, buffer size, and manual flushing for event delivery.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/configuring-how-events-are-sent/

When you [initialize your Tracker object](/docs/sources/ruby-tracker/getting-started/#tracking-design-and-initialization), you will need to provide one or more Emitter objects. Remember that we advise using the Singleton pattern, to avoid constantly recreating your objects.

There are two types of Emitter. The Emitter parent class can only send events synchronously. The AsyncEmitter subclass sends events asynchronously by default. We recommend you use the AsyncEmitter, to avoid slowing down your app.

Emitters and AsyncEmitters are initialized in the same way. An event collector endpoint is required, but there are various optional parameters that can also be set (see [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Emitter.html) and below table). A Tracker can have more than one associated Emitter or AsyncEmitter. These can be provided at Tracker initialization, or added later on.

Example:

**v0.7.0+:**

```ruby
emitter1 = SnowplowTracker::Emitter.new(endpoint: 'collector.example.com')

emitter2 = SnowplowTracker::AsyncEmitter.new(endpoint: 'collector2.example.com',

                                             options: { method: 'post' })

emitter3 = SnowplowTracker::AsyncEmitter.new(endpoint: 'collector.elsewhere.com',

                                             options: { protocol: 'https' })



tracker = SnowplowTracker::Tracker.new(emitters: [emitter1, emitter2])

tracker.add_emitter(emitter3)
```

**Before v0.7.0:**

```ruby
emitter1 = SnowplowTracker::Emitter.new('collector.example.com')

emitter2 = SnowplowTracker::AsyncEmitter.new('collector2.example.com',

                                             { method: 'post' })

emitter3 = SnowplowTracker::AsyncEmitter.new('collector.elsewhere.com',

                                             { protocol: 'https' })



tracker = SnowplowTracker::Tracker.new([emitter1, emitter2])

tracker.add_emitter(emitter3)
```

***

Optional Emitter settings:

| **Optional parameter** | **Description**                                                   | **Default** |
| ---------------------- | ----------------------------------------------------------------- | ----------- |
| `path`                 | Override the default path for appending to the endpoint           | n/a         |
| `protocol`             | `http` or `https`                                                 | `http`      |
| `port`                 | The port for the connection                                       | n/a         |
| `method`               | `get` or `post`                                                   | `get`       |
| `buffer_size`          | The size of the buffer, i.e. the number of events to send at once | 1           |
| `on_success`           | A method to call if events were all sent successfully             | n/a         |
| `on_failure`           | A method to call if any events did not send                       | n/a         |
| `thread_count`         | Number of threads to use (relevant to AsyncEmitters only)         | 1           |
| `logger`               | Where to log                                                      | STDERR      |

Response status codes of 2xx or 3xx status codes are considered successful.

### Manual flushing

You may want to force an emitter to send all events in its buffer, even if the buffer is not full. The `Tracker` class has a `flush` method which flushes all its emitters. It accepts one argument, `async`, which defaults to false. Unless you set `async` to `true`, the flush will be synchronous: it will block until all queued events have been sent. Of course, if you are using Emitters rather than AsyncEmitters, the flush will always be synchronous, even if `async` is set to `true`.

Example:

**v0.7.0+:**

```ruby
# Synchronous flush

tracker.flush



# Asynchronous flush

tracker.flush(async: true)
```

**Before v0.7.0:**

```ruby
# Synchronous flush

tracker.flush



# Asynchronous flush

tracker.flush(true)
```

***

### Logging

Emitters log messages about their activity and success during event sending. By default, Emitters log their activity to STDERR, using the Ruby standard library [Logger class](https://ruby-doc.org/stdlib-2.7.2/libdoc/logger/rdoc/Logger.html). Messages with a message level above INFO are logged. Advanced users might wish to alter how this logging occurs. You can change the message logging level, or, alternatively, you might want to disable logging completely, or log somewhere other than STDERR. Read how to do this in the [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Emitter.html).

---

# Get started with the Ruby tracker
> Install the Ruby tracker gem, initialize trackers with emitters, configure subject properties, and implement singleton pattern for Ruby and Rails apps.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/getting-started/

The Snowplow Ruby Tracker is compatible with Ruby 2.1+, including 3.0+. To add the Snowplow Tracker to your Ruby app or gem, add this line to your Gemfile:

```ruby
gem 'snowplow-tracker', '~> 0.8.0'
```

To make the Snowplow Ruby Tracker work with as many different Ruby programs as possible, we have tried to keep external dependencies to a minimum. There are only two external dependencies currently, both of which are for development of the gem itself: [rspec](https://rspec.info/) and [webmock](https://rubygems.org/gems/webmock).

## Introduction to the tracker code

Find the Ruby tracker code [here](https://github.com/snowplow/snowplow-ruby-tracker), and the API documentation [here](https://snowplow.github.io/snowplow-ruby-tracker/index.html).

The main class of the Ruby tracker is the Tracker class. Trackers provide methods for tracking events, such as `track_page_view`. The Tracker creates an appropriate event payload out of the provided event properties. This payload is passed to one or more Emitters for sending to the event collector.

Data about users, and which platform (e.g. server-side app or mobile) the event occurred on, are managed by Subject objects. A Subject can be added to each event, and Trackers always have an associated Subject. Other information can be added to events using Page, DeviceTimestamp or TrueTimestamp objects, as well as via event context. Event context is added via SelfDescribingJson objects.

> **Note:** The Ruby tracker version 0.7.0 had some breaking changes, notably the addition of keyword arguments. Check out the tracker changelog [here](https://github.com/snowplow/snowplow-ruby-tracker/blob/master/CHANGELOG).

## Tracking design and initialization

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

We suggest implementing the Ruby tracker as a Singleton global object. This pattern is demonstrated in our [Ruby on Rails example app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples). Structuring your code in this way avoids wasting bandwidth or processing on reinitializing Trackers and Emitters for every page load or event sent.

Note that the [Rails demo](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples) has both the Ruby and JavaScript trackers implemented. This combination of client- and server-side tracking can be highly effective and powerful. For example, you can discover how much effect adblockers are having on your tracking, by comparing the amount of client-side and server-side page view events you collect. It also allows you to track events in the most appropriate way for the event type. Check out this [blog post](https://snowplowanalytics.com/blog/2021/11/09/the-unrivaled-power-of-joining-client-and-server-side-tracking/) for more discussion about tracking client- and server-side.

The Tracker must be initialized with an Emitter or the subclass AsyncEmitter (or an array of Emitters/AsyncEmitters). The only required argument for an Emitter/AsyncEmitter is the endpoint, i.e. the address of the event collector.

Initializing a tracker instance:

**v0.7.0+:**

```ruby
require 'snowplow-tracker'



emitter = SnowplowTracker::Emitter.new(endpoint: 'collector.example.com')

tracker = SnowplowTracker::Tracker.new(emitters: emitter)

# or

tracker = SnowplowTracker::Tracker.new(emitters: [emitter])
```

**Before v0.7.0:**

```ruby
require 'snowplow-tracker'



emitter = SnowplowTracker::Emitter.new('collector.example.com')

tracker = SnowplowTracker::Tracker.new(emitter)

# or

tracker = SnowplowTracker::Tracker.new([emitter])
```

***

This Tracker will send events via GET to `http://collector.example.com/i`. To use other settings, such as POST or HTTPS, see "[Configuring how events are sent](/docs/sources/ruby-tracker/configuring-how-events-are-sent/)".

> **Note:** Do not include the protocol (HTTP or HTTPS) in your collector address. It will be added automatically.

At initialization, two Tracker parameters can be set which will be added to all events. The first is the Tracker `namespace`. This is especially useful to distinguish between events from different Trackers, if more than one is being used. The second user-set Tracker property is the `app_id`. This is the unique identifier for the site or application, and is particularly useful for distinguishing between events when Snowplow tracking has been implemented in multiple apps.

A Tracker is always associated with a Subject. It will be generated automatically if one is not provided during initialization. It can be swapped out later for another Subject using `set_subject`.

The final initialization parameter is a setting for the base64-encoding of any JSONs in the event payload. The default is for JSONs to be encoded. Once the Tracker has been instantiated, it is not possible to change this setting.

Initializing a tracker instance with all possible settings used:

**v0.7.0+:**

```ruby
require 'snowplow-tracker'



SnowplowTracker::Tracker.new(emitters: SnowplowTracker::Emitter.new(endpoint: 'collector.example.com'),

                             subject: SnowplowTracker::Subject.new,

                             namespace: 'tracker_no_encode',

                             app_id: 'rails_main',

                             encode_base64: false

                             )
```

**Before v0.7.0:**

```ruby
require 'snowplow-tracker'



SnowplowTracker::Tracker.new(SnowplowTracker::Emitter.new('collector.example.com'),

                             SnowplowTracker::Subject.new,

                             'tracker_no_encode',

                             'rails_main',

                             false

                             )
```

***

You can create as many Tracker instances as you like within your app; each one is completely sandboxed. For example, you may wish to send certain events to a different collector endpoint (using a different Emitter), or with a different JSON base64-encoding choice. Make sure you set different Tracker namespaces when using multiple instances.

## Testing your tracking

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

Check out our [Ruby tracker Rails demo](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples) and [Snowplow Micro examples repo](https://github.com/snowplow-incubator/snowplow-micro-examples) for two examples of automated webapp testing using Snowplow Micro and the testing framework [Cypress](https://www.cypress.io/). Any end-to-end testing framework can be used.

---

# Ruby tracker SDK
> Track events in Ruby applications, gems, and Rails web apps with the Snowplow Ruby tracker SDK compatible with Ruby 2.1+ and 3.0+.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/

The Snowplow Ruby Tracker allows you to track Snowplow events in your Ruby applications and gems and Ruby on Rails web applications.

The Ruby tracker is compatible with Ruby 2.1+, including Ruby 3.0+. The current tracker version is 0.8.0.

These pages will help you in setting up and using the Ruby tracker. There are more technical details in the [API docs.](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker.html)

As a server-side tracker, the Ruby tracker pairs very well with the JavaScript client-side tracker, if you are tracking events in a website. We've created a [simple demo Rails app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples), which incorporates both trackers, and also uses the Singleton pattern that we suggest for the Ruby tracker.

At its simplest, the Ruby tracker is used by calling a Tracker object's `track_x_event` method, such as `track_page_view`, which will send the event payload created to the Snowplow event collector.

There are three main classes which the Ruby Tracker uses: trackers, emitters, and subjects. These are introduced in more detail on the [next page](/docs/sources/ruby-tracker/getting-started/). There are also further classes which allow you to add extra data to your events, discussed on the page '[Adding data to your events: context and more](/docs/sources/ruby-tracker/adding-data-events/)'.

---

# Track events with the Ruby tracker
> Track self-describing events, structured events, page views, screen views, and ecommerce transactions with the Ruby tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/tracking-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your websites and apps.

We provide several built-in methods to help you track different kinds of events. The `track_x_event` methods range from single purpose methods, such as `track_page_view`, to the more complex but flexible `track_self_describing_event`, which can be used to track any kind of user behavior. We strongly recommend using `track_self_describing_event` for your tracking, as it allows you to design custom event types to match your business requirements. This [post on our blog](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/), "Re-thinking the structure of event data" might be informative here.

Tracking methods supported by the Ruby Tracker:

| **Method**                    | **Event type tracked**                               |
| ----------------------------- | ---------------------------------------------------- |
| `track_self_describing_event` | Custom event based on "self-describing" JSON schema  |
| `track_struct_event`          | Semi-custom structured event                         |
| `track_page_view`             | View of web page                                     |
| `track_screen_view`           | View of screen                                       |
| `track_ecommerce_transaction` | eCommerce transaction (and items in the transaction) |

All the `track_x_event` methods share common features and parameters. Every type of event can have an optional context, Subject, and Page added. A Timestamp can also be provided for all event types to override the default event timestamp. See [the next page](/docs/sources/ruby-tracker/adding-data-events/) to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event tracking methods, followed by the out-of-the-box event types. Note that you can also design and create your own page view, screen view, or eCommerce events, using `track_self_describing_event`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

### Track self-describing events with `track_self_describing_event`

Use `track_self_describing_event` to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the `data` value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

This method was previously called `track_unstruct_event`, as a counterpoint to `track_struct_event`. This name is misleading and we are in the process of depreciating it. An `unstruct` event requires a schema ruleset and therefore can be considered more structured than a `struct` event. However, this method still produces events labelled `unstruct`, as changing the event name in the Tracker Protocol would be a breaking change.

The `track_self_describing_event` method has one required argument, which must be a SelfDescribingJson object (see [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/SelfDescribingJson.html)). This class takes a schema name and a flat hash of event data. The keys of the hash can be either strings or Ruby symbols.

Example:

**v0.7.0+:**

```ruby
self_desc_json = SnowplowTracker::SelfDescribingJson.new(

  "iglu:com.example_company/save_game/jsonschema/1-0-2",

  {

    "saveId" => "4321",

    "level" => 23,

    "difficultyLevel" => "HARD",

    "dlContent" => true

  }

)



tracker.track_self_describing_event(event_json: self_desc_json)
```

**Before v0.7.0:**

```ruby
self_desc_json = SnowplowTracker::SelfDescribingJson.new(

  "iglu:com.example_company/save_game/jsonschema/1-0-2",

  {

    "saveId" => "4321",

    "level" => 23,

    "difficultyLevel" => "HARD",

    "dlContent" => true

  }

)



tracker.track_self_describing_event(self_desc_json)
```

***

You can track anything you want using this method, as long as you can describe it in a self-describing JSON schema.

### Track structured events with `track_struct_event`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category", "action", "label", and "value" properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| **Argument** | **Description**                                                  | **Required in event?** |
| ------------ | ---------------------------------------------------------------- | ---------------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes                    |
| `action`     | Defines the type of user interaction which this event involves   | Yes                    |
| `label`      | Often used to refer to the 'object' the action is performed on   | No                     |
| `property`   | Describing the 'object', or the action performed on it           | No                     |
| `value`      | Provides numerical data about the event                          | No                     |

Example:

**v0.7.0+:**

```ruby
tracker.track_struct_event(category: 'shop',

                           action: 'add-to-basket',

                           property: 'pcs',

                           value: 2)
```

**Before v0.7.0:**

```ruby
tracker.track_struct_event('shop', 'add-to-basket', nil, 'pcs', 2)
```

***

### Track page views with `track_page_view`

This is a simple, single-use method for tracking a user viewing a page within your app. You can record the page URL, the page title, and the referrer URL. Only the page URL is strictly required.

As a server-side language, your Ruby code won't automatically have access to the page title. This is one small reason why page views are easier to track client-side. Conversely, server-side page view tracking is more accurate, as it is not blocked by adblockers. It can be useful to compare counts from client- and server-side page views to see how much effect adblockers are having.

Example:

**v0.7.0+:**

```ruby
tracker.track_page_view(page_url: 'www.example.com',

                        page_title: 'example',

                        referrer: 'www.referrer.com')
```

**Before v0.7.0:**

```ruby
tracker.track_page_view('www.example.com', 'example', 'www.referrer.com')
```

***

### Track screen views with `track_screen_view`

Use `track_screen_view` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name` and `id`; while both are optional, you must provided at least one of them to create a valid event. "Name" is the human-readable screen name, and "ID" should be the unique screen ID.

This method creates an `unstruct` event, by creating a SelfDescribingJson and calling `track_self_describing_event`. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the name and/or ID which you provide. That schema is hosted on the schema repository [Iglu Central](http://iglucentral.com/), and so will always be available to your pipeline.

Example:

**v0.7.0+:**

```ruby
tracker.track_screen_view(name: 'HUD > Save Game',

                          id: 'screen23')
```

**Before v0.7.0:**

```ruby
tracker.track_screen_view('HUD > Save Game', 'screen23')
```

***

### Track eCommerce transactions with `track-ecommerce-transaction`

Use this out-of-the-box method to track an ecommerce transaction. This method is unique compared to the other `track_x_event` methods, as it sends multiple multiple events: one `transaction` event, and one `transaction_item` event for each item in the transaction. This is a legacy design; if we were creating a new eCommerce tracking method now, we would attach information about each item as event context entities to a single transaction event.

The arguments for this method are a "transaction" hash, and an array of "item" hashes. These hashes are strict about which keys are allowed and required. Check out the [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Tracker.html#track_ecommerce_transaction-instance_method) for the full details of the allowed properties. You can also read there about how additional event properties (context, Subject, Page and Timestamp) are handled for eCommerce events.

Broadly, the "transaction" hash contains the information about the order as a whole: the order ID, the value of the transaction including tax and/or shipping, as well as geographic information about its origin, or the currency in which the order was placed.

The "item" hash records each item's unique SKU identifier, value, and how many were purchased, as well as any further information which might be useful, such as its human-readable item name.

Example:

**v0.7.0+:**

```ruby
transaction = {

  order_id: '12345',

  total_value: 35,

  city: 'London',

  country: 'UK',

  currency: 'GBP'

}

item1 = {

  sku: 'pbz0026',

  price: 20,

  quantity: 1,

  category: 'film'

}

item2 = {

  sku: 'pbz0038',

  price: 15,

  quantity: 1,

  name: 'red shoes'

}

tracker.track_ecommerce_transaction(transaction: transaction,

                                    items: [item1, item2])
```

**Before v0.7.0:**

> **Note:** Older versions of the Ruby tracker have a bug in `track_ecommerce_transaction`: hashes with symbols for keys are not accepted. The event will fail silently. You must use only string keys. This was fixed in version 0.7.0.

```ruby
transaction = {

  'order_id' => '12345',

  'total_value' => 35,

  'city' => 'London',

  'country' => 'UK',

  'currency' => 'GBP'

}

item1 = {

  'sku' => 'pbz0026',

  'price' => 20,

  'quantity' => 1,

  'category' => 'film'

}

item2 = {

  'sku' => 'pbz0038',

  'price' => 15,

  'quantity' => 1,

  'name' => 'red shoes'

}

tracker.track_ecommerce_transaction(transaction, [item1, item2])
```

***

This will fire three events: one for the transaction as a whole, and one for each item. The `order_id` and `currency` fields in the `transaction` argument will also be attached to each of the item events.

---

# Add entities and other data to events with the Rust tracker
> Add event context entities and subject data with user and platform information to Rust tracker events using self-describing JSON schemas.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/adding-data/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset:

1. Event context using self-describing data: Attach event context, describing anything you like, in the form of self-describing JSONs.
2. Subject: Include information about the user, or the platform on which the event occurred.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event’s entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`context`) in your data storage. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

```rust
// Tracking a Self-Describing event with context entity

let self_describing_event = SelfDescribingEvent::builder()

    .schema("iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0")

    .data(json!({"name": "test", "id": "something else"}))

    .build()?;



let event_context = Some(vec![SelfDescribingJson::new(

    "iglu:org.schema/WebPage/jsonschema/1-0-0",

    json!({"keywords": ["tester"]}),

)]);



let self_desc_event_id = tracker.track(self_describing_event, event_context)?;
```

## Adding user and platform data with Subject

Subject information describes the user and device associated with the event. This information can be entered during tracker initialization by passing a `Subject` instance to the `Snowplow::create_tracker` method. All of the information is optional.

Some subject information is filled automatically by the tracker, such as the platform of the user, timezone, and language.

Please refer to the section on subject configuration on the [Configuration page](/docs/sources/rust-tracker/initialization-and-configuration/) to learn more.

---

# Get started with the Rust tracker
> Install the Rust tracker from crates.io, initialize with namespace and collector URL, and track screen view events with Tokio runtime.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/getting-started/

The following steps will guide you through setting up the Rust tracker in your project and tracking a simple event.

## Installation

Add the snowplow\_tracker as a dependency in Cargo.toml inside your Rust application:

```toml
[dependencies]

snowplow_tracker = 0.2.0
```

Use the package APIs in your code:

```rust
use snowplow_tracker::Snowplow;
```

## Initialization

Instantiate a tracker using the `Snowplow::create_tracker` function. The function takes three required arguments: `namespace`, `app_id`, `collector_url`, and one optional argument, `subject`. Tracker `namespace` identifies the tracker instance; you may create multiple trackers with different namespaces. The `app_id` identifies your app. The `collector_url` is the URI of the Snowplow collector to send the events to. `subject` allows for an optional Subject to be attached to the tracker, which will be sent with all events.

```rust
let mut tracker = Snowplow::create_tracker("ns", "app_id", "https://...", None);
```

There are additional optional arguments to configure the tracker. To learn more about configuring how events are sent, check out [this page](/docs/sources/rust-tracker/initialization-and-configuration/).

## Tracking events

To track events, simply instantiate their respective types and pass them to the tracker.track method with optional context entities. Please refer to the documentation for specification of event properties.

```rust
// Tracking a Screen View event

let screen_view_event = ScreenViewEvent::builder()

    .id(Uuid::new_v4())

    .name("a screen view")

    .previous_name("previous name")

    .build()?;



let screen_view_event_id = tracker.track(screen_view_event, None)?;
```

## Safely closing the emitter

The emitter spawns its own thread to send events using a Tokio runtime. To allow the tracker to drop, the emitter must be closed. This can be done by calling `Tracker.close_emitter()`.

```rust
let mut tracker = Snowplow::create_tracker("ns", "app_id", "https://...", None);

// track some events...

tracker.close_emitter();
```

Visit documentation about [tracking events](/docs/sources/rust-tracker/tracking-events/) to learn about other supported event types. You may also want to read about [adding more data to tracked events](/docs/sources/rust-tracker/adding-data/).

## Logging

The [log](https://crates.io/crates/log) crate is used for logging. To enable logging, you must add a logger to your application, such as [env\_logger](https://crates.io/crates/env_logger).

## Testing

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

---

# Rust tracker SDK
> Add event tracking to Rust applications using the Snowplow Rust tracker SDK with async event sending and retry policies.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/

The Snowplow Rust Tracker allows you to add analytics to your Rust apps.

The tracker is published on crates.io as [snowplow\_tracker](https://crates.io/crates/snowplow_tracker).

> **Note:** If you are upgrading from 0.1.0 to 0.2.0, there are a couple of changes required to make to your code:
>
> - [`Tracker.track`](/docs/sources/rust-tracker/getting-started/#tracking-events) is no longer an async function
> - The Emitter must be safely closed to allow the tracker to drop, as it spawns a thread to send events. This can be done by calling `Tracker.close_emitter()`.

---

# Initialize and configure the Rust tracker
> Configure Rust tracker with subject information, event store, HTTP client, batch emitter, and retry policies for async event delivery.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/initialization-and-configuration/

The `Snowplow` module provides a single method to initialize and configure a new tracker, the `Snowplow::create_tracker` method. It accepts configuration parameters for the tracker and returns a `Tracker` instance.

```rust
use snowplow_tracker::Subject;

let subject = Subject::builder().language("en-gb").build()?;

let tracker = Snowplow::create_tracker("ns", "app_id", "https://...", Some(subject));
```

The method returns a `Tracker` instance. This can be later used for tracking events, or accessing tracker properties.

The required attributes of the `Snowplow::create_tracker` method are `namespace` used to identify the tracker, `app_id` to identify your app, and the Snowplow collector `endpoint`. Additionally, one can pass in a `Subject` to provide additional context about the application environment.

| Attribute   | Type      | Description                                                                |
| ----------- | --------- | -------------------------------------------------------------------------- |
| `namespace` | `&str`    | Tracker namespace to identify the tracker.                                 |
| `app_id`    | `&str`    | Application identifier.                                                    |
| `endpoint`  | `&str`    | URI for the Snowplow collector endpoint.                                   |
| `subject`   | `Subject` | Subject information about tracked user and device that is added to events. |

## Configuration of subject information: `Subject`

Subject information are persistent and global information about the tracked device or user. They apply to all events and are assigned as event properties.

The configured attributes are mapped to Snowplow event properties described in the [Snowplow Tracker Protocol](/docs/events/). They are mapped as follows:

| Attribute         | Event Property   |
| ----------------- | ---------------- |
| `user_id`         | `uid`            |
| `network_user_id` | `network_userid` |
| `domain_user_id`  | `domain_userid`  |
| `user_agent`      | `useragent`      |
| `ip_address`      | `user_ipaddress` |
| `timezone`        | `os_timezone`    |
| `language`        | `lang`           |

## Configuring the Tracker

If you're looking for more control of your tracker, most components can be configured using their respective builder functions, or created from scratch by implementing the relevant trait.

## Event Store

An event store is defined by the `EventStore` trait. The default event store is the `InMemoryEventStore`. A custom event store can be created by implementing the `EventStore` trait, and passing it to the `BatchEmitter` builder.

### In Memory Event Store

The default event store for the tracker. This event store that keeps events in memory and does not persist them to disk. As `tracker.track` is called, events are added to the event store. When the event store has enough events to create an `EventBatch`, the events will be removed from the store, and sent to the collector.

The default capacity for the `InMemoryEventStore` is 10,000 events, and the default batch size is 50 events. An attempt to add an event to the store when it is full will result in the event being dropped, so choosing the right capacity is important. The capacity and batch size can be configured using the builder, for example, if we wanted to increase the capacity to 100,000 events and the batch size to 75 events:

```rust
let event_store = InMemoryEventStore::builder()

    .capacity(100_000)

    .batch_size(75)

    .build()?;
```

> **Warning:** Events in the `InMemoryEventStore` will be lost if the process is unexpectedly terminated.

## HTTP Client

A HTTP client is defined by the `HttpClient` trait. The default tracker HTTP client is the `ReqwestClient`, but, as with the Event Store, any implementation of the `HttpClient` trait can be passed to to the `BatchEmitter` builder:

```rust
let custom_http_client = MyCustomHttpClient::new();

let emitter = BatchEmitter::builder()

    .collector_url("http://localhost:9090")

    .http_client(custom_http_client)

    .build()?;
```

> **Note:** This is an async trait. The [async-trait](https://github.com/dtolnay/async-trait) crate is used until async traits are supported in stable Rust.

### Reqwest Client

The default HTTP client for the tracker. This client uses the [`reqwest`](https://github.com/seanmonstar/reqwest) crate to asynchronously send HTTP requests to the collector. The client only supports sending POST requests.

## Emitter

An emitter is defined by the `Emitter` trait, with the default emitter being the `BatchEmitter`. Any implementation of the `Emitter` trait can be passed to `Tracker::new`:

```rust
let custom_emitter = MyCustomEmitter::new();

let mut tracker = Tracker::new("ns", "app_id", custom_emitter, None);
```

### Batch Emitter

The BatchEmitter sends events in batches. This is more efficient than sending event requests singly, as only one set of POST headers is required for a number of events. The event collector receives the request and separates out the individual event payloads. Although the Snowplow Tracker interface is sync, the BatchEmitter uses [Tokio](https://github.com/tokio-rs/tokio) to asynchronously send events to the collector.

#### Retry Behavior

Failed requests are retried with exponentially increasing delays between subsequent retry requests. This ensures that the Collector is not overwhelmed with too many requests and also saves resources on the client by reducing the number of requests during failures. The number of times a request is retried is determined by the Retry Policy.

#### Retry Policy

A retry policy can be passed into the Emitter to define how many times a request should be retried.

The available policies are:

- `RetryForever` A RetryPolicy with no cap on maximum of attempts to send an event to the collector. This policy might appear attractive where it is critical to avoid data loss because it never deliberately drops events. However, it could cause a backlog of events in the buffered queue if the collector is unavailable for too long.
- `MaxAttempts(max: u32)` A RetryPolicy which drops events after failing to contact the collector within a fixed number of attempts. This policy can smooth over short outages of connection to the collector. Events will be dropped only if the collector is unreachable for a relatively long span of time. Dropping events can be a safety mechanism against a growing backlog of unsent events.
- `NoRetry` A RetryPolicy that drops events immediately after a failed attempt to send to the collector.

The default retry policy on the `BatchEmitter` is `RetryPolicy::MaxAttempts(10)`, but can be configured using the builder, for example if we wanted to disable retries:

```rust
let emitter = BatchEmitter::builder()

    .collector_url("http://localhost:9090")

    .retry_policy(RetryPolicy::NoRetry)

    .build()?;
```

#### Safely Stopping the Emitter

The `BatchEmitter` can be safely stopped using the `tracker.close_emitter()` method. This will stop the emitter from accepting new events, and will wait for any in-flight requests to complete before returning. This is useful as the emitter runs in a separate thread, and the main thread needs to wait for the emitter to finish sending events before exiting.

---

# Track events with the Rust tracker
> Track self-describing events, structured events, screen views, and timing events with the Rust tracker SDK using builder patterns.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/tracking-events/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in events to help you track different kinds of events. When instantiated, their objects can be passed to the `tracker.track()` method to send events to the Snowplow collector. The events range from single purpose ones, such as `ScreenViewEvent`, to the more complex but flexible `SelfDescribingEvent`, which can be used to track any kind of user behavior. We strongly recommend using `SelfDescribingEvent` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

Event classes supported by the Rust Tracker:

| Method                | Event type tracked                                          |
| --------------------- | ----------------------------------------------------------- |
| `SelfDescribingEvent` | Custom event based on "self-describing" JSON schema         |
| `StructuredEvent`     | Semi-custom structured event                                |
| `ScreenViewEvent`     | View of a screen in the app                                 |
| `TimingEvent`         | User timing events such as how long resources take to load. |

All the methods share common features and parameters. Every type of event can have an optional context added. See the [next page](/docs/sources/rust-tracker/adding-data/) to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event types, followed by the out-of-the-box event types. Note that you can also design and create your own page view, or screen view, using `SelfDescribingEvent`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

## Track self-describing events with `SelfDescribingEvent`

Use the `SelfDescribingEvent` type to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

Example (assumes that `tracker` is a tracker instance created using `Snowplow::create_tracker`):

```rust
// Tracking a Self-Describing event with context entity

let self_describing_event = SelfDescribingEvent::builder()

    .schema("iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0")

    .data(json!({"name": "test", "id": "something else"}))

    .build()?;



let event_context = Some(vec![SelfDescribingJson::new(

    "iglu:org.schema/WebPage/jsonschema/1-0-0",

    json!({"keywords": ["tester"]}),

)]);



let self_desc_event_id = tracker.track(self_describing_event, event_context)?;
```

## Track structured events with `StructuredEvent`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category", "action", "label", and "value" properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| Argument   | Description                                                    | Required in event? |
| ---------- | -------------------------------------------------------------- | ------------------ |
| `category` | The grouping of structured events which this action belongs to | Yes                |
| `action`   | Defines the type of user interaction which this event involves | Yes                |
| `label`    | Often used to refer to the 'object' the action is performed on | No                 |
| `property` | Describing the 'object', or the action performed on it         | No                 |
| `value`    | Provides numerical data about the event                        | No                 |

Example:

```rust
// Tracking a Structured event

let structured_event = StructuredEvent::builder()

    .category("shop")

    .action("add-to-basket")

    .label("Add To Basket")

    .property("pcs")

    .value(2.0)

    .build()?;



let struct_event_id = tracker.track(structured_event, None)?;
```

## Track screen views with `ScreenViewEvent`

Use `ScreenViewEvent` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name`, `id`, `type`, and `transitionType`. The `name` and `id` properties are required. "Name" is the human-readable screen name, and "ID" should be the unique screen ID (UUID v4).

This method creates an unstruct event, by creating and tracking a self-describing event. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the parameters which you provide. That schema is hosted on the schema repository Iglu Central, and so will always be available to your pipeline.

| Argument         | Description                                                 | Required in event? |
| ---------------- | ----------------------------------------------------------- | ------------------ |
| `name`           | The name of the screen viewed.                              | Yes                |
| `id`             | The id (UUID v4) of screen that was viewed.                 | Yes                |
| `type`           | The type of screen that was viewed.                         | No                 |
| `previousName`   | The name of the previous screen that was viewed.            | No                 |
| `previousType`   | The type of screen that was viewed.                         | No                 |
| `previousId`     | The id (UUID v4) of the previous screen that was viewed.    | No                 |
| `transitionType` | The type of transition that led to the screen being viewed. | No                 |

Example:

```rust
// Tracking a Screen View event

let screen_view_event = ScreenViewEvent::builder()

    .id(Uuid::new_v4())

    .name("a screen view")

    .previous_name("previous name")

    .build()?;



let screen_view_event_id = tracker.track(screen_view_event, None)?;
```

## Track timing events with `TimingEvent`

Use the `TimingEvent` type to track user timing events such as how long resources take to load. These events take a timing `category`, the `variable` being measured, and the `TimingEvent` time measurement. An optional `label` can be added to further identify the timing event

| Argument   | Description                                              | Required in event? |
| ---------- | -------------------------------------------------------- | ------------------ |
| `category` | Defines the timing category.                             | Yes                |
| `variable` | Defines the timing variable measured.                    | Yes                |
| `timing`   | Represents the time.                                     | Yes                |
| `label`    | An optional string to further identify the timing event. | No                 |

Example:

```rust
let timing_event = TimingEvent::builder()

    .category("fetch_resource")

    .variable("map_loaded")

    .timing(1423)

    .label("Time to fetch map resource")

    .build()?;



let timing_event_id = tracker.track(timing_event, None)?;
```

---

# Scala tracker SDK
> Track events in Scala applications and servers with subjects, emitters, and async event sending using the Snowplow Scala tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/

The Snowplow Scala Tracker allows you to track Snowplow events in your Scala apps and servers. The tracker should be straightforward to use if you are comfortable with Scala development.

There are three main classes which the Scala Tracker uses: subjects, emitters, and trackers.

A subject represents a single user whose events are tracked, and holds data specific to that user.

A tracker always has one active subject at a time associated with it. The default subject only has "platform=server" configured, but you can replace it with a subject containing more data. The tracker constructs events with that subject and sends them to one or more emitters, which sends them on to a Snowplow collector.

---

# Initialize the Scala tracker
> Initialize Scala tracker with ID or http4s emitters, configure buffer size, retry policies, event queue policies, and add callbacks for event sending.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/initialization/

Assuming you have completed the Scala Tracker Setup, you are ready to initialize the Scala Tracker.

## Emitters

Each tracker instance must be initialized with an Emitter which is responsible for firing events to a Collector. We offer two different modules of emitter: the http4s-emitter and the id-emitter

### ID Emitters

The emitters in the "id" module have a simpler developer interface, because return types are not wrapped in functional events. Use these emitters if your application code is not using any other typeclasses from the cats ecosystem.

The below code:

- Creates a non-blocking emitter `emitter1` which sends events to "mycollector.com" on the default port 80
- Creates a blocking emitter `emitter2` which sends events to “myothercollector.com” on port 8080
- Creates a tracker which can be used to send events to all emitters

```scala
import scala.concurrent.ExecutionContext.Implicits.global



import com.snowplowanalytics.snowplow.scalatracker.Tracker

import com.snowplowanalytics.snowplow.scalatracker.Emitter._

import com.snowplowanalytics.snowplow.scalatracker.emitters.id._

import cats.data.NonEmptyList



val emitter1 = AsyncEmitter.createAndStart(EndpointParams("mycollector.com"))

val emitter2 = SyncEmitter(EndpointParams("myothercollector.com", port = Some(8080)))

val tracker = new Tracker(NonEmptyList.of(emitter1, emitter2), "mytrackername", "myapplicationid")
```

When using this emitter, the methods on the tracker all return `Unit`, so there is no requirement to flatMap over effect types.

```scala
tracker.trackPageView("http://example.com") // returns Unit

tracker.trackTransaction("order1234", 42.0) // returns Unit
```

The `SyncEmitter` blocks the whole thread when it sends events to the collector. The `AsyncEmitter` sends requests asynchronously from the tracker's main thread of execution, but in doing so it blocks a thread on the provided execution for each http request. The blocking calls are wrapped in scala's [blocking construct](https://www.scala-lang.org/api/current/scala/concurrent/index.html#blocking%5BT%5D\(body:=%3ET\):T), which is respected by the global execution context.

We reccomend using `AsyncEmitter` if possible as to not block threads, only using `SyncEmitter` if you have a specific need for sync in your setup.

### Http4s Emitter

Backed by a [Http4s](https://http4s.org/) client, this emitter captures actions in the context of a functional effect type, such as a cats IO, ZIO or Monix Task. This is the recommended emitter if you are familiar with functional programming and the cats ecosystem of type classes.

```scala
import cats.data.NonEmptyList

import cats.effect.std.Random

import cats.effect.{IO, Resource}

import org.http4s.ember.client.EmberClientBuilder



import com.snowplowanalytics.snowplow.scalatracker._

import com.snowplowanalytics.snowplow.scalatracker.Emitter._

import com.snowplowanalytics.snowplow.scalatracker.emitters.http4s._



val resource: Resource[IO, Tracker[IO]] = for {

  client <- EmberClientBuilder.default[IO].build

  rng <- Resource.eval(Random.scalaUtilRandom[IO])

  emitter1 <- Http4sEmitter.build[IO](EndpointParams("mycollector.com"), client)(implicitly, rng)

  emitter2 <- Http4sEmitter.build[IO](EndpointParams("myothercollector.com", port = Some(8080)), client)(implicitly, rng)

} yield new Tracker(NonEmptyList.of(emitter1, emitter2), "mytrackername", "myapplicationid")



resource.use { tracker =>

  // Use the tracker inside this block to initialize and run your application

  MyApp.run(tracker)

}
```

The above code:

- Creates a Http4s client from the EmberClientBuilder. Http4s offers other variants of the client, but Ember is the recommended default choice.
- Creates a Random instance, required to randomize the delay on request retries
- Creates an emitter `emitter1` which sends events to "mycollector.com" on the default port 80
- Creates a second emitter `emitter2` which sends events to “myothercollector.com” on port 8080
- Creates a tracker which can be used to send events to all emitters

Your application might then use the tracker by flatMapping over the functional effect:

```scala
for {

  _ <- tracker.trackPageView("http://example.com")

  _ <- tracker.trackTransaction("order1234", 42.0)

} yield ()
```

## Subject

You can configure a subject with extra data and attach it to the tracker so that the data will be attached to every event:

```scala
val subject = new Subject()

  .setUserId("user-00035")

  .setPlatform(Desktop)

val tracker = Tracker(emitters, ns, appId).setSubject(subject)
```

Alternatively, you can set the subject for each event individually.

## Buffer Configuration

Emitters use a buffer, so they can send larger payloads comprising several events, instead of sending each event immediately. You can choose the behavior of your emitter's buffering by passing in a `BufferConfig` during initialisation. For example:

```scala
val emitter = Http4sEmitter.build[IO](endpoint, bufferConfig = PayloadSize(40000))
```

The available configs are:

- `EventsCardinality(size: Int)` Configures the emitter to buffer events and send batched payloads comprising a fixed number of events
- `PayloadSize(bytes: Int)` Configures the emitter to buffer events and send batched payloads of a minimum size in bytes
- `NoBuffering` Configures the emitter to send events immediately to the collector, without buffering for larger batches
- `OneOf(left: BufferConfig, right: BufferConfig)` Configures the emitter to buffer events until either of the inner BufferConfigs say the queue is full

## Retry Policies

Emitters can be configured to retry sending events to the collector if the initial attempt fails. For example:

```scala
val emitter = Http4sEmitter.build[IO](endpoint, retryPolicy = MaxAttempts(10))
```

The available policies are:

- `RetryForever` A RetryPolicy with no cap on maximum of attempts to send an event to the collector. This policy might appear attractive where it is critical to avoid data loss because it never deliberately drops events. However, it could cause a backlog of events in the buffered queue if the collector is unavailable for too long. This RetryPolicy could be paired with an `EventQueuePolicy` that manages the behavior of a large backlog.
- `MaxAttempts(max: Int)` A RetryPolicy which drops events after failing to contact the collector within a fixed number of attempts. This policy can smooth over short outages of connection to the collector. Events will be dropped only if the collector is unreachable for a relatively long span of time. Dropping events can be a safety mechanism against a growing backlog of unsent events.
- `NoRetry` A RetryPolicy that drops events immediately after a failed attempt to send to the collector.

## Event Queue Policy

An `EventQueuePolicy` becomes important when the queue of pending events grows to an unexpectedly large number; for example, if the collector is unreachable for a long period of time

Picking an EventQueuePolicy is an opportunity to protect against excessive heap usage by limiting the maximum size of the queue

An EventQueuePolicy can be paired with an appropriate `RetryPolicy`, which controls dropping events when they cannot be sent

For example:

```scala
val emitter = new Http4sEmitter(endpoint, queuePolicy = IgnoreWhenFull(100))
```

The available policies are:

- `UnboundedQueue`. An `EventQueuePolicy` with no upper bound on the number pending events in the emitter's queue. This policy never deliberately drops events, but it comes at the expense of potentially high heap usage if the collector is unavailable for a long period of time.
- `BlockwhenFull(limit: Int)`. An `EventQueuePolicy` that blocks the tracker's thread until the queue of pending events falls below a threshold. This policy never deliberately drops events, but it comes at the expense of blocking threads if the collector is unavailable for long period of time.
- `IgnoreWhenFull(limit: Int)`. An `EventQueuePolicy` that silently drops new events when the queue of pending events exceeds a threshold.
- `ErrorWhenFull(limit: Int)`. An `EventQueuePolicy` that raises an exception when the queue of pending events exceeds a threshold.

## Global contexts

You can configure your tracker to add a context to every event sent:

```scala
val tracker = Tracker(emitters, namespace, appId).addContext(selfDescribingJson)
```

There is also syntax for adding EC2 or GCE contexts automatically:

#### EC2 Context

Amazon [Elastic Cloud](https://aws.amazon.com/ec2/) can provide basic information about instance running your app. You can add this informational as additional custom context to all sent events by enabling it in Tracker after initializaiton of your tracker:

```scala
import com.snowplowanalytics.snowplow.scalatracker.metadata._

val tracker = Tracker(emitters, ns, appId).enableEc2Context()
```

#### Google Compute Engine Metadata context

Google [Cloud Compute Engine](https://cloud.google.com/compute) can provide basic information about instance running your app. You can add this informational as additional custom context to all sent events by enabling it in Tracker after initializaiton of your tracker:

```scala
import com.snowplowanalytics.snowplow.scalatracker.metadata._

val tracker = Tracker(emitters, ns, appId).enableGceContext()
```

This will add [`iglu:com.google.cloud.gce/instance_metadata/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/152c90a72d5888460985ea43605afb5252180b10/schemas/com.google.cloud.gce/instance_metadata/jsonschema/1-0-0) context to all your events

## Callbacks

All emitters supplied with Scala Tracker support callbacks invoked after every sent event (or batch of events) whether it was successful or not. This feature particularly useful for checking collector unavailability and tracker debugging.

Callbacks should have following signature:

```scala
type Callback = (Emitter.EndpointParams, Emitter.Request, Emitter.Result) => Unit
```

- `EndpointParams` is collector configuration attached to emitter
- `Request` is raw collector's payload, which can be either `GET` or `POST` and holding number of undertaken attempts
- `Result` is processed collector's response or failure reason. You'll want to pattern-match it to either no-op or notify DevOps about non-working collector

To add a callback to `AsyncBatchEmitter` you can use following approach:

```scala
import com.snowplowanalytics.snowplow.scalatracker.Emitter._



def emitterCallback(params: EndpointParams, req: Request, res: Result): Unit = {

  res match {

    case Result.Success(_) => ()

    case Result.Failure(code) =>

      devopsIncident(s"Scala Tracker got unexpected HTTP code $code from ${params.getUri}")

    case Result.TrackerFailure(exception) =>

      devopsIncident(s"Scala Tracker failed to reach ${params.getUri} with following exception $exception after ${req.attempt} attempt")

    case Result.RetriesExceeded(failure) =>

      devopsIncident(s"Scala Tracker has stopped trying to deliver payload after following failure: $failure")

      savePayload(req)      // can be investigated and sent afterwards

  }

}



val emitter = AsyncEmitter.createAndStart(endpointParams, callback = Some(emitterCallback))
```

The async emitter will perform callbacks asynchronously in its `ExecutionContext`.

---

# Track events with the Scala tracker
> Track self-describing events, structured events, page views, errors, and ecommerce transactions with the Scala tracker using custom contexts and timestamps.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/sending-events/

This page lists the available methods for sending events to Snowplow.

## `trackSelfDescribingEvent`

Use `trackSelfDescribingEvent` to track a custom Self-describing events (previously known as Unstructured Events) which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

You can use its alias `trackUnstructEvent`.

| **Argument**        | **Description**                                    | **Required?** | **Type**                   |
| ------------------- | -------------------------------------------------- | ------------- | -------------------------- |
| `unstructuredEvent` | Self-describing JSON containing unstructured event | Yes           | `SelfDescribingJson`       |
| `contexts`          | List of custom contexts for the event              | No            | `List[SelfDescribingJson]` |
| `timestamp`         | Device created timestamp or true timestamp         | No            | `Option[Timestamp]`        |

Create a Snowplow unstructured event [self-describing JSON](https://github.com/snowplow/iglu/wiki/Self-describing-JSONs):

```scala
import com.snowplowanalytics.iglu.core.{SchemaKey, SchemaVer}

import io.circe.Json



val productViewEvent = SelfDescribingJson(

  SchemaKey("com.acme", "product_view", "jsonschema", SchemaVer(1,0,0)),

  Json.obj(

    "sku" := "0000013"

  )

)
```

Send it using the `trackSelfDescribingEvent` tracker method:

```scala
tracker.trackSelfDescribingEvent(productViewEvent)
```

You can attach any number of custom contexts to an event:

```scala
val pageTypeContext = SelfDescribingJson(

  SchemaKey("com.acme", "page_type", "jsonschema", SchemaVer(1,0,0)),

  Json.obj(

    "type" := "promotional",

    "backgroundColor" := "red"

  )

)



val userContext = SelfDescribingJson(

  SchemaKey("com.acme", "user", "jsonschema", SchemaVer(1,0,0)),

  Json.obj(

    "userType" := "tester"

  )

)



t.trackSelfDescribingEvent(productViewEvent, List(pageTypeContext, userContext))
```

## `trackStructEvent`

Use `trackStructEvent` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required).

| **Argument** | **Description**                                                  | **Required?** | **Type**                   |
| ------------ | ---------------------------------------------------------------- | ------------- | -------------------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes           | `String`                   |
| `action`     | Defines the type of user interaction which this event involves   | Yes           | `String`                   |
| `label`      | A string to provide additional dimensions to the event data      | No            | `Option[String]`           |
| `property`   | A string describing the object or the action performed on it     | No            | `Option[String]`           |
| `value`      | A value to provide numerical data about the event                | No            | `Option[Double]`           |
| `contexts`   | List of custom contexts for the event                            | No            | `List[SelfDescribingJson]` |
| `timestamp`  | Device created timestamp or true timestamp                       | No            | `Option[Timestamp]`        |

Example:

```scala
val pageTypeContext = SelfDescribingJson(

  SchemaKey("com.acme", "page_type", "jsonschema", SchemaVer(1,0,0)),

  Json.obj(

    "type" := "promotional",

    "backgroundColor" := "red"

  )

)



val userContext = SelfDescribingJson(

  SchemaKey("com.acme", "user", "jsonschema", SchemaVer(1,0,0)),

  Json.obj(

    "userType" := "tester"

  )

)



t.trackStructEvent("commerce", "order", property=Some("book"), contexts=List(pageTypeContext, userContext))
```

## `trackPageView`

Use `trackPageView` to track a user viewing a page within your app. Arguments are:

| **Argument** | **Description**                      | **Required?** | **Validation**             |
| ------------ | ------------------------------------ | ------------- | -------------------------- |
| `pageUrl`    | The URL of the page                  | Yes           | `String`                   |
| `pageTitle`  | The title of the page                | No            | `Option[String]`           |
| `referrer`   | The address which linked to the page | No            | `Option[String]`           |
| `contexts`   | Custom contexts for the event        | No            | `List[SelfDescribingJson]` |
| `timestamp`  | When the pageview occurred           | No            | `Option[Timestamp]`        |

Example:

```scala
t.trackPageView("www.example.com", Some("example"), Some("www.referrer.com"))
```

## `trackError`

Use `trackError` to track exceptions raised during your app's execution. Arguments are:

| **Argument** | **Description**                  | **Required?** | **Validation**             |
| ------------ | -------------------------------- | ------------- | -------------------------- |
| `error`      | Any throwable need to be tracked | Yes           | `Throwable`                |
| `contexts`   | Custom contexts for the event    | No            | `List[SelfDescribingJson]` |
| `timestamp`  | When the pageview occurred       | No            | `Option[Timestamp]`        |

Example:

```scala
try {

  1 / 0

} catch {

  case e: java.lang.ArithmeticException =>

    t.trackError(e)

}
```

Note: this tracker should not be used to track exceptions happening in tracker itself, use [callbacks](/docs/sources/scala-tracker/initialization/#callbacks) mechanism for that.

## `trackAddToCart`

Use `trackAddToCart` to track an add-to-cart event.

## `trackRemoveFromCart`

Use `trackRemoveFromCart` to track a remove-from-cart event.

## `trackTransaction`

Use `trackTransaction` to record view of transaction. Fire a `trackTransaction` to register the transaction, and then fire `trackTransactionItem` to log specific data about the items that were part of that transaction.

## `trackTransactionItem`

To track an ecommerce transaction item. Fire a `trackTransaction` to register the transaction, and then fire `trackTransactionItem` to log specific data about the items that were part of that transaction.

## Setting timestamp

By default, Scala Tracker will generate a `dvce_created_tstamp` and add it to event payload. You also can manually set it using `timestamp` argument in all tracking methods. It should be in milliseconds since the Unix epoch:

```scala
tracker.trackSelfDescribingEvent(productViewEvent, Nil, Some(1432806619000L))
```

Beside of it, you can set `true_tstamp` if you have more reliable source about event timestamp. You can tag timstamp as "true" using class `TrueTimestamp`:

```scala
tracker.trackSelfDescribingEvent(productViewEvent, Nil, Some(Tracker.TrueTimestamp(1432806619000L)))
```

Now event will be sent with `ttm` parameter instead of `dtm`.

## Setting the subject

By default, each event is associated with the (optional) `Subject` defined on the tracker instance. You can also send a `Subject` for each individual event, which overrides the default Subject.

```scala
val subject = new Subject()

  .setUserId("user-00035")

  .setPlatform(Desktop)

t.trackPageView("www.example.com", subject = Some(subject))
```

---

# Install the Scala tracker
> Install the Snowplow Scala tracker from Maven Central or JCenter using sbt, Gradle, or Maven with support for http4s and id emitters.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/setup/

The Tracker is published to Maven Central and JCenter, which should make it easy to add it as a dependency into your own Scala app.

**sbt:**

Add the Scala Tracker to your build.sbt:

```scala
libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-core" % "2.0.0"

// If you plan to use the http4s emitter with an Ember client

libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-emitter-http4s" % "2.0.0"

libraryDependencies += "org.http4s" %% "http4s-ember-client" % "0.23.15"

// If you plan to use the id emitters:

libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-emitter-id" % "2.0.0"

// If you plan to use EC2/GCE contexts:

libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-metadata" % "2.0.0"
```

**Gradle:**

Add our Maven repository in your `build.gradle` file:

```gradle
repositories {

  ...

  jcenter()

}
```

Then add into the same file:

```gradle
dependencies {

  ...

  // Snowplow Scala Tracker

  compile 'com.snowplowanalytics:snowplow-scala-tracker-core_2.13:2.0.0'

  // If you plan to use the http4s emitters with an Ember client

  compile 'com.snowplowanalytics:snowplow-scala-tracker-emitter-http4s_2.13:2.0.0'

  compile 'org.http4s:http4s-ember-client_2.13:0.23.15'

  // If you plan to use the id emitters

  compile 'com.snowplowanalytics:snowplow-scala-tracker-emitter-id_2.13:2.0.0'

  // If you plan to use EC2/GCE contexts:

  compile 'com.snowplowanalytics:snowplow-scala-tracker-metadata_2.13:2.0.0'

}
```

**Maven:**

Add into your project's `pom.xml`:

```maven
<dependency>

  <groupId>com.snowplowanalytics</groupId>

  <artifactId>snowplow-scala-tracker-core_2.13</artifactId>

  <version>2.0.0</version>

</dependency>

<!-- If you plan to use the http4s emitter with an Ember client: -->

<dependency>

  <groupId>com.snowplowanalytics</groupId>

  <artifactId>snowplow-scala-tracker-emitter-http4s_2.13</artifactId>

  <version>2.0.0</version>

</dependency>

<dependency>

  <groupId>org.http4s</groupId>

  <artifactId>http4s-ember-client</artifactId>

  <version>0.23.15</version>

</dependency>

<!-- If you plan to use the id emitters: -->

<dependency>

  <groupId>com.snowplowanalytics</groupId>

  <artifactId>snowplow-scala-tracker-emitter-id_2.13</artifactId>

  <version>2.0.0</version>

</dependency>

<!-- If you plan to use EC2/GCE contexts: -->

<dependency>

  <groupId>com.snowplowanalytics</groupId>

  <artifactId>snowplow-scala-tracker-metadata_2.13</artifactId>

  <version>2.0.0</version>

</dependency>
```

***

> **Info:** Notice a `_2.13` postfix in artifactId. This is used for Scala libraries and denotes the Scala version which the artifact (in our case `snowplow-scala-tracker`) is compiled against. It also means that this library will bring a `org.scala-lang:scala-library_2.13.x` as transitive dependency and if you're using any other Scala dependency you should keep these postfixes in accordance (`snowplow-scala-tracker` is also compiled against Scala 2.12).

---

# Configure a subject with the Scala tracker
> Configure Scala tracker subject with platform, user ID, screen resolution, viewport, timezone, language, IP address, and domain user ID properties.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/subject-methods/

A list of the methods used to add data to the Subject class.

### Set the platform with `setPlatform`

The default platform is `Server`. These are the available alternatives, all available in the package `com.snowplowanalytics.snowplow.scalatracker`:

- Server
- Web
- Mobile
- Desktop
- Tv
- Console
- InternetOfThings
- General

Example usage

```scala
subject.setPlatform(Tv)
```

### Set the user ID with `setUserId`

You can make the user ID a string of your choice:

```scala
subject.setUserId("user-000563456")
```

### Set the screen resolution with `setScreenResolution`

If your Scala code has access to the device's screen resolution, you can pass it in to Snowplow. Both numbers should be positive integers; note the order is width followed by height. Example:

```scala
subject.setScreenResolution(1366, 768)
```

### Set the viewport dimensions with `setViewport`

Similarly, you can pass the viewport dimensions in to Snowplow. Again, both numbers should be positive integers and the order is width followed by height. Example:

```scala
subject.setViewport(300, 200)
```

### Set the color depth with `setColorDepth`

If your Scala code has access to the bit depth of the device's color palette for displaying images, you can pass it in to Snowplow. The number should be a positive integer, in bits per pixel.

```scala
subject.setColorDepth(24)
```

### Setting the timezone with `setTimezone`

If your Scala code has access to the timezone of the device, you can pass it in to Snowplow:

```scala
subject.setTimezone("Europe London")
```

### Setting the language with `setLang`

You can set the language field like this:

```scala
subject.setLang("en")
```

### Setting the IP address with `setIpAddress`

If you have access to the user's IP address, you can set it like this:

```scala
subject.setIpAddresss("34.634.11.139")
```

### Setting the useragent with `setUseragent`

If you have access to the user's useragent (sometimes called "browser string"), you can set it like this:

```scala
subject.setUseragent("Mozilla/5.0 (Windows NT 5.1; rv:24.0) Gecko/20100101 Firefox/24.0")
```

### Setting the domain user ID with `setDomainUserId`

The `domain_userid` field of the Snowplow event model corresponds to the ID stored in the first party cookie set by the Snowplow JavaScript Tracker. If you want to match up server-side events with client-side events, you can set the domain user ID for server-side events like this:

```scala
subject.setDomainUserId("c7aadf5c60a5dff9")
```

### Setting the network user ID with `setNetworkUserId`

The `network_user_id` field of the Snowplow event model corresponds to the ID stored in the third party cookie set by the Snowplow Clojure Collector and Scala Stream Collector. You can set the network user ID for server-side events like this:

```scala
subject.setNetworkUserId("ecdff4d0-9175-40ac-a8bb-325c49733607")
```

---

# Additional information for the Tracking CLI
> Learn about Tracking CLI implementation details including Golang tracker backend, no buffering behavior, and collector response exit codes.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/additional-information/

There is no buffering in the Snowplow Tracking CLI - each event is sent as an individual payload whether `GET` or `POST`.

Under the hood, the app uses the [**Snowplow Golang Tracker**](https://github.com/snowplow/snowplow-golang-tracker).

The Snowplow Tracking CLI will exit once the Snowplow collector has responded. The return codes from `snowplowtrk` are as follows:

- 0 if the Snowplow collector responded with an OK status (2xx or 3xx)
- 4 if the Snowplow collector responded with a 4xx status
- 5 if the Snowplow collector responded with a 5xx status
- 1 for any other error

---

# Tracking CLI
> Deprecated. The Snowplow Tracking CLI has been merged into Snowplow CLI. Use the snowplow-cli events send command to send events from the command line instead.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/

> **Warning:** The standalone Snowplow Tracking CLI has been merged into [Snowplow CLI](/docs/api-reference/snowplow-cli/) as the `events send` command. This tool is deprecated and its repository will be archived. See [Send events via Snowplow CLI](/docs/api-reference/snowplow-cli/send-events/) for the replacement and a migration guide.

The Snowplow Tracking CLI is a native app to make it easy to send an event to Snowplow from the command line. Use this to embed Snowplow tracking into your shell scripts and terminal sessions.

---

# Install the Snowplow Tracking CLI
> Download Snowplow Tracking CLI binaries for Linux, macOS, and Windows, or use the Docker container for command-line event tracking.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/installation/

You can download the binary for Linux, macOS and Windows directly from GitHub:

- [**Linux AMD 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_linux_amd64.zip)
- [**Windows AMD 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_windows_amd64.zip)
- [**macOS AMD 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_darwin_amd64.zip)
- [**macOS ARM 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_darwin_arm64.zip)

There is also a [Docker Container available](https://hub.docker.com/r/snowplow/snowplow-tracking-cli) which can be used in lieu of installing the binary directly.

---

# Snowplow Tracking CLI interface
> Deprecated. Reference for the legacy Snowplow Tracking CLI interface. Use the snowplow-cli events send command instead.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/usage/

The command line interface is as follows:

```bash
snowplow-tracking-cli --collector={{COLLECTOR_DOMAIN}} --appid={{APP_ID}} --method=[POST|GET] --sdjson={{SELF_DESC_JSON}}
```

or:

```bash
snowplow-tracking-cli --collector={{COLLECTOR_DOMAIN}} --appid={{APP_ID}} --method=[POST|GET] --schema={{SCHEMA_URI}} --json={{JSON}}
```

where:

- `--collector` is the domain for your Snowplow collector, e.g. `snowplow-collector.acme.com`
- `--appid` is optional (not sent if not set)
- `--method` is optional. It defaults to `GET`
- `--protocol` is optional. It defaults to `https`
- `--sdjson` is a self-describing JSON of the standard form `{ "schema": "iglu:...", "data": { ... } }`
- `--schema` is a schema URI, most likely of the form `iglu:...`
- `--json` is a (non-self-describing) JSON, of the form `{ ... }`
- `--ipaddress` is optional. It defaults to an empty string
- `--contexts` is optional. It defaults to an empty JSON array `[]`

The idea here is that you can either send in a [**self-describing JSON**](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/), or pass in the constituent parts (i.e. a regular JSON plus a schema URI) and the Snowplow Tracking CLI will construct the final self-describing JSON for you.

## Examples

```bash
snowplow-tracking-cli --collector snowplow-collector.acme.com --appid myappid --method POST --schema iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0 --json "{\"hello\":\"world\"}"
```

```bash
snowplow-tracking-cli --collector snowplow-collector.acme.com --appid myappid --method POST --sdjson "{\"schema\":\"iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0\", \"data\":{\"hello\":\"world\"}}"
```

## Usage with Docker

```bash
docker run snowplow/snowplow-tracking-cli:latest --collector snowplow-collector.acme.com --appid myappid --method POST --sdjson "{\"schema\":\"iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0\", \"data\":{\"hello\":\"world\"}}"
```

---

# Tracker maintenance classification
> Understand Snowplow tracker maintenance groups including actively maintained, maintained, early release, and unsupported status with production readiness and update schedules.
> Source: https://docs.snowplow.io/docs/sources/tracker-maintenance-classification/

Snowplow Trackers and dbt packages have been categorized into 4 maintenance groups. These groups are designed to give some expectations into how they will maintained in the future. Each tracker GitHub repository contains a "Snowplow" Badge that shows the maintenance group for that specific tracker and links back to this page.

***

Not production-ready, not actively maintained. Deprecated.

***

Pre-Version 1.0.0, may be missing core tracker features. Accepting community contributions. Snowplow Team will test and release contributions. Snowplow Team will update to fix critical or high severity vulnerabilities. Will be released less frequently.

***

Currently production ready. Maintained by the Snowplow team but released less frequently. Accepting community contributions. May not have up to date dependencies or support for latest versions of language or framework. Scanned daily for vulnerabilities, Snowplow Team will fix critical or high severity vulnerabilities promptly.

***

Currently production ready. Actively maintained by the Snowplow team, with a frequent release schedule. Accepting community contributions. Dependencies are actively updated and will support the latest versions of language or framework. Scanned daily for vulnerabilities, Snowplow Team will fix all vulnerabilities promptly.

***

| Status | Production Ready | Release Frequency         | Accepting Contributions | Dependency and Vulnerability Updates                                                                            |
| ------ | ---------------- | ------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------- |
|        | ❌                | Never                     | ❌                       | None                                                                                                            |
|        | Varies           | As required               | ✔️                      | - Updates for critical and high severity vulnerabilities                                                        |
|        | ✔️               | As required Low frequency | ✔️                      | - Daily scans - Passive updates to dependencies - Active updates for critical and high severity vulnerabilities |
|        | ✔️               | Frequent release schedule | ✔️                      | - Daily scans - Active updates to dependencies and all levels of vulnerabilities                                |

---

# Configure emitters in the Unity tracker
> Configure sync, async, and WebGL emitters to send and store Unity tracker events with customizable protocols, methods, and batch sizes.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/emitter/

The Emitter object is responsible for sending and storing all events.

We provide the following emitters available currently:

- `SyncEmitter` : Slow blocking synchronous operation, useful for testing but should not be used in production.
- `AsyncEmitter` : Fully asynchronous operation which uses the ThreadPool to perform all of its operations.
- `WebGlEmitter` : Emitter implementing the async API that is compatible with WebGL.

### Constructor

| **Argument Name** | **Description**                                             | **Required?** | **Default**                                                                                      |
| ----------------- | ----------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------ |
| `endpoint`        | The collector uri the emitter should use                    | Yes           | Null                                                                                             |
| `protocol`        | The request Protocol (HTTP or HTTPS)                        | No            | HTTP                                                                                             |
| `method`          | The HTTP Method (GET or POST)                               | No            | POST                                                                                             |
| `sendLimit`       | The amount of events to send at a time                      | No            | 500 (AsyncEmitter) / 10 (SyncEmitter) / 1 (WebGlEmitter)                                         |
| `byteLimitGet`    | The byte limit for a GET request                            | No            | 52000                                                                                            |
| `byteLimitPost`   | The byte limit for a POST request                           | No            | 52000                                                                                            |
| `eventStore`      | Implementation for storing queued events waiting to be sent | No            | `InMemoryEventStore` on tvOS, otherwise `EventStore` (on-disk [LiteDB](https://www.litedb.org/)) |

A full Emitter construction should look like the following:

```csharp
IEmitter e1 = new AsyncEmitter ("com.collector.acme", HttpProtocol.HTTPS, HttpMethod.GET, 50, 30000, 30000, null);
```

All except `eventStore` of these variables can be altered after creation with the accompanying `emitter.SetXXX()` function. However do be aware that multiple threads will be accessing these variables so to be safe always shut the Tracker down using `StopEventTracking()` before amending anything.

**NOTE:** Be aware that when sending events via `GET` all events will be sent individually. This means that if your `sendLimit` is 500 there is the potential for 500 Threads to be spawned at the same time which can cause serious performance issues. To alleviate this concern simply drop your `sendLimit` to a more manageable range.

Some load balancers limit the maximum URL length accepted for `GET` requests, often around 16KB. The default of 52000 bytes is considerably larger than these limits, but most single event payloads will not reach this size.

### Functions

The Emitter also provides functions to configure its behavior after construction.

#### `SetServerAnonymisation(bool)`

Enables or disables [server-side anonymization](/docs/events/anonymous-tracking/#server-side-anonymization). When enabled, the `SP-Anonymous: *` header is added to all requests to the Collector, instructing it to anonymize the user's IP address and network user ID.

```csharp
emitter.SetServerAnonymisation(true);
```

---

# Track events with the Unity tracker
> Track page views, screen views, structured events, timing events, self-describing events, and ecommerce transactions in Unity applications.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/event-tracking/

You track events by building an event payload using event-specific Builder objects, and passing the result to a tracker instance's `Track()` method. Your events can have common fields configured with the respective `SetCustomContexts()`, `SetTimestamp()`, and `SetEventId()` methods, in addition to event-specific fields covered below.

## Track page views with `Track(PageView)`

You can use `Track(PageView)` to track a user viewing a web page within your app.

Arguments are:

| **Argument**     | **Description**                      | **Required?** | **Type**         |
| ---------------- | ------------------------------------ | ------------- | ---------------- |
| `pageUrl`        | The URL of the page                  | Yes           | `string`         |
| `pageTitle`      | The title of the page                | No            | `string`         |
| `referrer`       | The address which linked to the page | No            | `string`         |
| `customContexts` | Optional custom context              | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                   | No            | `long`           |
| `eventId`        | Optional custom event id             | No            | `string`         |

Examples:

```csharp
t1.Track(new PageView()

    .SetPageUrl("www.example.com")

    .SetPageTitle("example")

    .SetReferrer("www.referrer.com")

    .Build());



t1.Track(new PageView()

    .SetPageUrl("www.example.com")

    .SetPageTitle("example")

    .SetReferrer("www.referrer.com")

    .SetCustomContext(contextList)

    .SetTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

## Track screen views with `Track(MobileScreenView)`

Use `Track(MobileScreenView)` to track a user viewing a screen (or equivalent) within your app. You **must** provide a `name` property. The `id` of the screen view will be automatically assigned (as an UUID) but you may also provide it manually. Arguments are:

| **Argument**     | **Description**                                                | **Required?** | **Type**         |
| ---------------- | -------------------------------------------------------------- | ------------- | ---------------- |
| `name`           | Human-readable name for this screen                            | Yes           | `string`         |
| `id`             | Unique identifier for this screen view (can be auto-generated) | Yes           | `string`         |
| `type`           | The type of screen that was viewed e.g feed / carousel         | No            | `string`         |
| `previousName`   | The name of the previous screen                                | No            | `string`         |
| `previousId`     | The screen view ID of the previous screen view                 | No            | `string`         |
| `previousType`   | The screen type of the previous screen view                    | No            | `string`         |
| `previousType`   | The screen type of the previous screen view                    | No            | `string`         |
| `customContexts` | Optional custom context                                        | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                                             | No            | `long`           |
| `eventId`        | Optional custom event id                                       | No            | `string`         |

Examples:

```csharp
t1.Track(new MobileScreenView("HUD > Save Game")

    .SetType("dialog")

    .SetPreviousName("HUD > Menu")

    .SetPreviousId(previousScreenId)

    .SetTransitionType("pop-up")

    .Build());



t1.Track(new MobileScreenView("HUD > Save Game")

    .SetCustomContext(contextList)

    .SetTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

> **Note:** In tracker versions 0.7.0 and earlier, screen views were tracked using the `ScreenView` class. However, this class used the older schema for screen views and has been deprecated in favour of the `MobileScreenView` in the 0.8.0 release of the tracker.

## Track structured events with `Track(Structured)`

Use `Track(Structured)` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument**     | **Description**                                                  | **Required?** | **Type**         |
| ---------------- | ---------------------------------------------------------------- | ------------- | ---------------- |
| `category`       | The grouping of structured events which this `action` belongs to | Yes           | `string`         |
| `action`         | Defines the type of user interaction which this event involves   | Yes           | `string`         |
| `label`          | A string to provide additional dimensions to the event data      | No            | `string`         |
| `property`       | A string describing the object or the action performed on it     | No            | `string`         |
| `value`          | A value to provide numerical data about the event                | No            | `double`         |
| `customContexts` | Optional custom context                                          | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                                               | No            | `long`           |
| `eventId`        | Optional custom event id                                         | No            | `string`         |

Examples:

```csharp
t1.Track(new Structured()

    .SetCategory("shop")

    .SetAction("add-to-basket")

    .Build());



t1.Track(new Structured()

    .SetCategory("shop")

    .SetAction("add-to-basket")

    .SetLabel("Add To Basket")

    .SetProperty("pcs")

    .SetValue(2.00)

    .SetCustomContext(contextList)

    .SetTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

## Track timing events with `Track(Timing)`

Use `Track(Timing)` to track an event related to a custom timing.

| **Argument**     | **Description**                        | **Required?** | **Type**         |
| ---------------- | -------------------------------------- | ------------- | ---------------- |
| `category`       | The category of the timed event        | Yes           | `string`         |
| `label`          | The label of the timed event           | No            | `string`         |
| `timing`         | The timing measurement in milliseconds | Yes           | `int`            |
| `variable`       | The name of the timed event            | Yes           | `string`         |
| `customContexts` | Optional custom context                | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                     | No            | `long`           |
| `eventId`        | Optional custom event id               | No            | `string`         |

Examples:

```csharp
t1.Track(new Timing()

    .SetCategory("category")

    .SetVariable("variable")

    .SetTiming(1)

    .Build());



t1.Track(new Timing()

    .SetCategory("category")

    .SetVariable("variable")

    .SetTiming(1)

    .SetLabel("label")

    .SetCustomContext(contextList)

    .SetTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build());
```

## Track self-describing events with `Track(SelfDescribing)`

Custom self-describing events are a flexible tool that enable Snowplow users to define their own event types and send them into Snowplow.

When a user sends in a custom self-describing event, they do so as a JSON of name-value properties, that conforms to a JSON schema defined for the event earlier.

Use `Track(SelfDescribing)` to track a custom event which consists of a name and a self-describing set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

The arguments are as follows:

| **Argument**     | **Description**             | **Required?** | **Type**                                    |
| ---------------- | --------------------------- | ------------- | ------------------------------------------- |
| `eventData`      | The properties of the event | Yes           | [`SelfDescribingJson`](#selfdescribingjson) |
| `customContexts` | Optional custom context     | No            | `List<IContext>`                            |
| `timestamp`      | Optional timestamp          | No            | `long`                                      |
| `eventId`        | Optional custom event id    | No            | `string`                                    |

Example event json to track:

```json
{

  "schema": "iglu:com.acme/save_game/jsonschema/1-0-0",

  "data": {

    "levelName": "Barrels o' Fun",

    "levelIndex": 23

  }

}
```

How to set it up?

```json
// Create a Dictionary of your event data

Dictionary<string, object> eventDict = new Dictionary<string, object>();

eventDict.Add("levelName", "Barrels o' Fun");

eventDict.Add("levelIndex", 23);



// Track your event with your custom event data

t1.Track(new SelfDescribing("iglu:com.acme/save_game/jsonschema/1-0-0", eventDict)

    .Build();



// OR



t1.Track(new SelfDescribing("iglu:com.acme/save_game/jsonschema/1-0-0", eventDict)

    .SetCustomContext(contextList)

    .SetTimestamp(1423583655000)

    .SetEventId("uid-1")

    .Build();
```

For more on JSON schema, see the [blog post](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

> **Note:** In versions 0.7.0 and earlier, the `Unstructured` class was used to track self-describing events. This was deprecated in version 0.8.0 in favour of the `SelfDescribing` class.

## Track ecommerce transactions with `Track(EcommerceTransaction)`

Use `Track(EcommerceTransaction)` to track an ecommerce transaction.

Arguments:

| **Argument**     | **Description**                 | **Required?** | **Type**                         |
| ---------------- | ------------------------------- | ------------- | -------------------------------- |
| `orderId`        | ID of the eCommerce transaction | Yes           | `string`                         |
| `totalValue`     | Total transaction value         | Yes           | `double`                         |
| `affiliation`    | Transaction affiliation         | No            | `string`                         |
| `taxValue`       | Transaction tax value           | No            | `double`                         |
| `shipping`       | Delivery cost charged           | No            | `double`                         |
| `city`           | Delivery address city           | No            | `string`                         |
| `state`          | Delivery address state          | No            | `string`                         |
| `country`        | Delivery address country        | No            | `string`                         |
| `currency`       | Transaction currency            | No            | `string`                         |
| `items`          | Items in the transaction        | Yes           | `List<EcommerceTransactionItem>` |
| `customContexts` | Optional custom context         | No            | `List<IContext>`                 |
| `timestamp`      | Optional timestamp              | No            | `long`                           |
| `eventId`        | Optional custom event id        | No            | `string`                         |

The `items` argument is a `List` of individual `EcommerceTransactionItem` elements representing the items in the ecommerce transaction. Note that `Track(EcommerceTransaction)` fires multiple events: one transaction event for the transaction as a whole, and one transaction item event for each element of the `items` `List`.

Each transaction item event will have the same timestamp, orderId, and currency as the main transaction event.

#### `EcommerceTransactionItem`

To instantiate a `EcommerceTransactionItem` in your code, simply use the following constructor signature:

```json
EcommerceTransactionItem item = new EcommerceTransactionItem ()

      .SetSku ("sku")

      .SetPrice (10.2)

      .SetQuantity (1)

      .SetName ("name")

      .SetCategory ("category")

      .Build ()
```

These are the fields that can appear as elements in each `EcommerceTransactionItem` element of the transaction item's `List`:

| **Field**        | **Description**          | **Required?** | **Type**         |
| ---------------- | ------------------------ | ------------- | ---------------- |
| `sku`            | Item SKU                 | Yes           | `string`         |
| `price`          | Item price               | Yes           | `double`         |
| `quantity`       | Item quantity            | Yes           | `int`            |
| `name`           | Item name                | No            | `string`         |
| `category`       | Item category            | No            | `string`         |
| `customContexts` | Optional custom context  | No            | `List<IContext>` |
| `eventId`        | Optional custom event id | No            | `string`         |

Example of tracking a transaction containing two items:

```json
// Create some Transaction Items

EcommerceTransactionItem item1 = new EcommerceTransactionItem ()

    .SetSku ("item_sku_1")

    .SetPrice (10.2)

    .SetQuantity (1)

    .SetName ("item_name_1")

    .SetCategory ("item_category")

    .Build ();



EcommerceTransactionItem item2 = new EcommerceTransactionItem()

    .SetSku("item_sku_2")

    .SetPrice(1.00)

    .SetQuantity(1)

    .SetName("item_name_2")

    .SetCategory("item_category")

    .Build();



// Add these items to a List

List<EcommerceTransactionItem> items = new List<EcommerceTransactionItem>();

items.Add(item1);

items.Add(item2);



// Now Track the Transaction by using this list of items as an argument

tracker.Track(new EcommerceTransaction()

    .SetOrderId("order_id_1")

    .SetTotalValue(300.00)

    .SetAffiliation("my_affiliate")

    .SetTaxValue(30.00)

    .SetShipping(10.00)

    .SetCity("Boston")

    .SetState("Massachusetts")

    .SetCountry("USA")

    .SetCurrency("USD")

    .SetItems(items)

    .Build());
```

## Custom Contexts

Custom contexts are Self Describing Jsons with extra descriptive information that can be optionally attached to any Snowplow event with `SetCustomContexts(...)`. We provide several builders for Snowplow custom contexts as well as a generic builder if you wish to define and send your own custom context!

For ease of development you are also able to extend the `IContext` interface or the `AbstractContext` class for your own contexts if you so wish.

All of these contexts will need to be combined into a `List<IContext>` before being attachable to Snowplow Events.

### `DesktopContext`

The following arguments can be used in a DesktopContext:

| **Field**            | **Description**                     | **Required?** | **Type** |
| -------------------- | ----------------------------------- | ------------- | -------- |
| `osType`             | The Operating System Type           | Yes           | `string` |
| `osVersion`          | The Version of the Operating System | Yes           | `string` |
| `osServicePack`      | Service Pack information            | No            | `string` |
| `osIs64Bit`          | If the OS is 32 or 64 bit           | No            | `bool`   |
| `deviceManufacturer` | Who made the device                 | No            | `string` |
| `deviceModel`        | What is the device model            | No            | `string` |
| `processorCount`     | How many cores does the device have | No            | `int`    |

An example of a DesktopContext construction:

```json
DesktopContext context = new DesktopContext ()

    .SetOsType("OS-X")

    .SetOsVersion("10.10.5")

    .SetOsServicePack("Yosemite")

    .SetOsIs64Bit(true)

    .SetDeviceManufacturer("Apple")

    .SetDeviceModel("Macbook Pro")

    .SetDeviceProcessorCount(4)

    .Build ();
```

### `MobileContext`

The following arguments can be used in a MobileContext:

| **Field**            | **Description**                     | **Required?** | **Type**      |
| -------------------- | ----------------------------------- | ------------- | ------------- |
| `osType`             | The Operating System Type           | Yes           | `string`      |
| `osVersion`          | The Version of the Operating System | Yes           | `string`      |
| `deviceManufacturer` | Who made the device                 | Yes           | `string`      |
| `deviceModel`        | What is the device model            | Yes           | `string`      |
| `carrier`            | The name of the carrier             | No            | `string`      |
| `networkType`        | The type of network                 | No            | `NetworkType` |
| `networkTechnology`  | The networks technlogy              | No            | `string`      |
| `openIdfa`           | An OpenIDFA UUID                    | No            | `string`      |
| `appleIdfa`          | An Apple IDFA UUID                  | No            | `string`      |
| `appleIdfv`          | An Apple IDFV UUID                  | No            | `string`      |
| `androidIdfa`        | An Android IDFA UUID                | No            | `string`      |

An example of a MobileContext construction:

```json
MobileContext context = new MobileContext ()

    .SetOsType("iOS")

    .SetOsVersion("9.0")

    .SetDeviceManufacturer("Apple")

    .SetDeviceModel("iPhone 6S+")

    .SetCarrier("FREE")

    .SetNetworkType(NetworkType.Mobile)

    .SetNetworkTechnology("LTE")

    .Build ();
```

### `GeoLocationContext`

The following arguments can be used in a GeoLocationContext:

| **Field**                   | **Description**            | **Required?** | **Type** |
| --------------------------- | -------------------------- | ------------- | -------- |
| `latitude`                  | The user latitude          | Yes           | `double` |
| `longitude`                 | The user longitude         | Yes           | `double` |
| `latitudeLongitudeAccuracy` | The user lat-long accuracy | No            | `double` |
| `altitude`                  | The user altitude          | No            | `double` |
| `altitudeAccuracy`          | The user alt accuracy      | No            | `double` |
| `bearing`                   | The user bearing           | No            | `double` |
| `speed`                     | The user speed             | No            | `double` |
| `timestamp`                 | A timestamp in ms          | No            | `long`   |

An example of a GeoLocationContext construction:

```json
GeoLocationContext context = new GeoLocationContext ()

    .SetLatitude(123.564)

    .SetLongitude(-12.6)

    .SetLatitudeLongitudeAccuracy(5.6)

    .SetAltitude(5.5)

    .SetAltitudeAccuracy(2.1)

    .SetBearing(3.2)

    .SetSpeed(100.2)

    .SetTimestamp(1234567890000)

    .Build ();
```

### `GenericContext`

The GenericContext is a simple builder with three functions:

- `SetSchema(string)` : Sets the Context Schema Path
- `Add(string, object)` : Adds a single key-pair value to the data packet of this context
- `AddDict(string, object)` : Adds a dictionary of key-pair values to the data packet

You must set a schema string or a RuntimeException will be thrown.

An example of a GenericContext construction:

```json
GenericContext context = new GenericContext()

    .SetSchema("iglu:com.acme/acme_context/jsonschema/1-0-0")

    .Add("context", "custom")

    .Build();
```

## SelfDescribingJson

A `SelfDescribingJson` is used as a wrapper around a `Dictionary<string, object>`. After creating the Dictionary you want to wrap you can create a `SelfDescribingJson` using the following:

```json
// Data as a Dictionary

Dictionary<string, object> data = new Dictionary<string, object>();

data.Add("Event", "Data")



// We then create a new SelfDescribingJson

SelfDescribingJson json = new SelfDescribingJson("iglu:com.acme/example/jsonschema/1-0-0", data);
```

This object is now ready to be Tracked within a SelfDescribing Event.

You can create a SelfDescribingJson with the following arguments:

| **Argument** | **Description**                           | **Required?** | **Type**                    |
| ------------ | ----------------------------------------- | ------------- | --------------------------- |
| `schema`     | JsonSchema that describes the data        | Yes           | `string`                    |
| `data`       | Data that will be validated by the schema | No            | `Dictionary<string,object`> |

---

# Unity tracker SDK
> Track Snowplow events from Unity games and apps across multiple platforms including iOS, Android, WebGL, and desktop.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/

The Snowplow Unity Tracker allows you to track Snowplow events from your Unity games and apps. The current tracker version is 0.8.1.

### Snowplow Icebreaker Demo

To see the Tracker in action you can download our demonstration game Snowplow Icebreaker from [Github](https://github.com/snowplow/snowplow-unity-tracker) and loading the SnowplowTracker.Demo folder into Unity 2022.3. You will need to set a valid collector endpoint on the Main Menu Page to see the events being generated.

---

# Initialize the Unity tracker
> Import required libraries and create tracker instances with emitters in Unity projects across platforms including WebGL.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/initialization/

Assuming you have completed the Unity Tracker Setup for your project, you are now ready to initialize the Unity Tracker.

### Importing the library

Add the following `using` lines to the top of your `.cs` scripts to access the Tracker:

```csharp
using SnowplowTracker;

using SnowplowTracker.Emitters;

using SnowplowTracker.Enums;

using SnowplowTracker.Events;
```

You should now be able to setup the Tracker!

### Creating a tracker

To instantiate a Tracker in your code (can be global or local to the process being tracked) simply instantiate the Tracker interface with the following:

```csharp
// Create Emitter – to support WebGL, make sure to use the WebGlEmitter

IEmitter emitter = Application.platform == RuntimePlatform.WebGLPlayer

        ? new WebGlEmitter("com.collector.acme", HttpProtocol.HTTPS, HttpMethod.POST)

        : new AsyncEmitter("com.collector.acme", HttpProtocol.HTTPS, HttpMethod.POST);



// Create a Tracker instance with the Emitter

Tracker tracker = new Tracker(emitter, "Namespace", "AppId");



// Start the Tracker

tracker.StartEventTracking();
```

---

# Track session data with the Unity tracker
> Manage user session data with configurable foreground and background timeouts across Unity platforms with persistent storage.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/session/

The Session object is responsible for maintaining persistent data around user sessions over the life-time of an application.

Session information is persisted using file storage on most platforms except for tvOS and WebGL. On those two platforms, session information is stored in [`PlayerPrefs`](https://docs.unity3d.com/ScriptReference/PlayerPrefs.html).

### Constructor

| **Argument Name**   | **Description**                                        | **Required?**      | **Default**              |
| ------------------- | ------------------------------------------------------ | ------------------ | ------------------------ |
| `sessionPath`       | The file path to store the persistent session state in | Yes (accepts null) | "snowplow\_session.dict" |
| `foregroundTimeout` | The time until a session expires in focus              | No                 | 600 (s)                  |
| `backgroundTimeout` | The time until a session expires in back               | No                 | 300 (s)                  |
| `checkInterval`     | How often to validate the session timeout              | No                 | 15 (s)                   |

A full Session construction should look like the following:

```csharp
Session session = new Session ("snowplow_session_state.json", 1200, 600, 30);
```

A minimal construction would be:

```csharp
Session session = new Session (null);
```

The timeouts refer to the length of time the session remains active after the last event is sent. As long as events are sent within this limit the session will not timeout.

### Functions

Except `sessionPath`, all of the constructor variables can be altered/retrieved after creation with the accompanying `session.SetXXX()`/`session.GetXXX()` functions.

#### `SetBackground(bool)`

Will set whether or not the application is in the background. It is up to the developer to set this metric if they wish to have a different timeout for foreground and background.

---

# Install the Unity tracker
> Install the Unity tracker package for Unity 2022.3+ supporting Windows, macOS, Linux, iOS, Android, tvOS, and WebGL platforms.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/setup/

The Snowplow Unity Tracker has been built and tested using Unity 2022.3 on Windows, Linux, OSX, iOS, Android, tvOS, and WebGL. It is built using .NET Standard 2.0 and requires at least Unity 2018.1.

It is not currently compatible with the WebPlayer.

## Dependencies

There are several dependencies that are required to use the Unity Tracker currently. These are available in [this folder](https://github.com/snowplow/snowplow-unity-tracker/tree/master/Resources).

- `UnityEngine.dll` : This is included as a way of setting up development externally to the Unity playground; you do not need to include this.
- `Newtonsoft.Json.dll` : This is included as a way of setting up development externally to the Unity playground; you do not need to include this as it is bundled with Unity 2020+ or should be installed via [Newtonsoft.Json-for-Unity package](https://github.com/jilleJr/Newtonsoft.Json-for-Unity).

All the other dependencies are required for proper Tracker function but they are included inside the `unitypackage`.

## Download

The Unity Tracker package is published to the Unity Trackers [Github releases](https://github.com/snowplow/snowplow-unity-tracker/releases). Once downloaded simply add the `SnowplowTracker.unitypackage` to your project. This should insert all of the required dll files into your `Assets/Plugins/` folder.

---

# Configure a subject with the Unity tracker
> Configure subject properties including user ID, screen resolution, viewport, timezone, and language to attach user context to all tracked events.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/subject/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

> **Note:** When [user anonymization](/docs/sources/unity-tracker/tracker/#setuseranonymisationbool) is enabled on the tracker, the `userId`, `domainUserId`, `networkUserId`, and `ipAddress` fields are omitted from events regardless of what is set on the Subject.

The Subject class has a set of `Set...()` methods to attach extra data relating to the user to all tracked events:

- `SetUserId`
- `SetScreenResolution`
- `SetViewport`
- `SetColorDepth`
- `SetTimezone`
- `SetLanguage`
- `SetIpAddress`
- `SetUseragent`
- `SetNetworkUserId`
- `SetDomainUserId`

Here are some examples:

```csharp
Subject s1 = new Subject();

s1.SetUserId("Kevin Gleason");

s1.SetLanguage("en-gb");

s1.SetScreenResolution(1920, 1080);
```

After that, you can add your Subject to your Tracker like so:

```csharp
Tracker t1 = new Tracker(..., s1)



// OR



t1.SetSubject(s1);
```

### Set user ID with `SetUserId`

You can set the user ID to any string:

```csharp
s1.SetUserId( "{{USER ID}}" )
```

Example:

```csharp
s1.SetUserId("alexd")
```

### Set screen resolution with `SetScreenResolution`

If your C# code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```csharp
t1.SetScreenResolution( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
t1.SetScreenResolution(1366, 768)
```

### Set viewport dimensions with `SetViewport`

If your C# code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```csharp
s.SetViewport( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
s.SetViewport(300, 200)
```

### Set color depth with `SetColorDepth`

If your C# code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```csharp
s.SetColorDepth( {{BITS PER PIXEL}} )
```

The number should be a positive integer, in bits per pixel. Example:

```csharp
s.SetColorDepth(32)
```

### Set timezone with `SetTimezone`

This method lets you pass a user's timezone in to Snowplow:

```csharp
s.SetTimezone( {{TIMEZONE}} )
```

The timezone should be a string:

```csharp
s.SetTimezone("Europe/London")
```

### Set the language with `SetLanguage`

This method lets you pass a user's language in to Snowplow:

```csharp
s.SetLanguage( {{LANGUAGE}} )
```

The language should be a string:

```csharp
s.SetLanguage('en')
```

### `SetIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```csharp
SetIpAddress( {{IP ADDRESS}} )
```

The IP address should be a string:

```csharp
subj.SetIpAddress("127.0.0.1");
```

### `SetUseragent`

This method lets you pass a useragent in to Snowplow:

```csharp
SetUseragent( {{USERAGENT}} )
```

The useragent should be a string:

```csharp
subj.SetUseragent("Agent Smith");
```

### `SetNetworkUserId`

This method lets you pass a Network User ID in to Snowplow:

```csharp
SetNetworkUserId( {{NUID}} )
```

The network user id should be a string:

```csharp
subj.SetNetworkUserId("network-id");
```

### `SetDomainUserId`

This method lets you pass a Domain User ID in to Snowplow:

```csharp
SetDomainUserId( {{DUID}} )
```

The domain user id should be a string:

```csharp
subj.SetDomainUserId("domain-id");
```

---

# Configure a tracker with the Unity tracker
> Configure the Unity tracker instance to coordinate event tracking, manage sessions, and control background processing with customizable platform settings.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/tracker/

The Tracker object is responsible for co-ordinating the saving and sending of events as well as managing the optional Session object.

### Constructor

| **Argument Name**                                 | **Description**                                                                                                                                                                                                                                                                                                                                               | **Required?** | **Default** |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------- |
| [`emitter`](/docs/sources/unity-tracker/emitter/) | The Emitter object you create                                                                                                                                                                                                                                                                                                                                 | Yes           | Null        |
| `trackerNamespace`                                | The name of the tracker instance                                                                                                                                                                                                                                                                                                                              | Yes           | Null        |
| `appId`                                           | The application ID                                                                                                                                                                                                                                                                                                                                            | Yes           | Null        |
| [`subject`](/docs/sources/unity-tracker/subject/) | The Subject that defines a user                                                                                                                                                                                                                                                                                                                               | No            | Null        |
| [`session`](/docs/sources/unity-tracker/session/) | The Session object you create                                                                                                                                                                                                                                                                                                                                 | No            | Null        |
| `platform`                                        | The device the Tracker is running on                                                                                                                                                                                                                                                                                                                          | No            | Mobile      |
| `base64Encoded`                                   | If we [base 64 encode](https://en.wikipedia.org/wiki/Base64) json values                                                                                                                                                                                                                                                                                      | No            | True        |
| `userAnonymisation`                               | Enables [client-side user anonymization](/docs/events/anonymous-tracking/#full-client-side-anonymization). Omits user identifiers (userId, domainUserId, networkUserId) and any explicitly-set ipAddress from all events. Note: this does not affect the IP address captured by the collector from the network connection, use server anonymisation for that. | No            | False       |

A full Tracker construction should look like the following:

```csharp
IEmitter e1 = new AsyncEmitter ("com.collector.acme")

Subject subject = new Subject();

Session session = new Session(null);

Tracker t1 = new Tracker(e1, "Namespace", "AppId", subject, session, DevicePlatforms.Desktop, true);
```

All of these variables can be altered after creation with the accompanying `tracker.SetXXX()` function.

### Functions

The Tracker also contains several critical functions that must be used to start Tracking.

#### `StartEventTracking()`

This function must be called before any events will start being stored or sent. This is due to the fact that we do not want to start any background processing from the constructors so it is left up to the developer to choose when to start everything up.

This function:

- Starts the background emitter thread
- Starts the background event processor thread
- Starts the background session check timer (Optional)

Once this is run everything should be in place for asynchronous event tracking.

#### `StopEventTracking()`

If you need to temporarily halt the Tracker from tracking events you can run this function. This will bring all event processing, sending and collection to a halt and nothing will be started again until `StartEventTracking()` is fired again.

#### `Track(IEvent)`

This is the function used for Tracking all events.

```csharp
tracker.Track(IEvent newEvent);
```

#### `SetUserAnonymisation(bool)`

Enables or disables user anonymization. When enabled, the `userId`, `domainUserId`, `networkUserId`, and `ipAddress` fields are omitted from all events. This can also be set at construction time via the `userAnonymisation` parameter.

```csharp
tracker.SetUserAnonymisation(true);
```

---

# Unity tracker utilities
> Use logging utilities and thread-safe queue implementations for debugging and managing Unity tracker operations.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/utilities/

The Tracker also provides several extra utilities that can be used.

## Log

There is a custom logging wrapper which allows you to control the level of Tracker logging that occurs. You can set the level via:

```csharp
Log.SetLogLevel({0,1,2 or 3});
```

- `0` : Turns of all logging
- `1` : Error only
- `2` : Error and Debug
- `3` : Error, Debug and Verbose

## 8.2 ConcurrentQueue

Due to the .NET 2.0 limitations in earlier versions of the Snowplow Unity Tracker we have had to implement our own ThreadSafe queue which can be found in the following package:

- `SnowplowTracker.Collections`

---

# Opt-outs and anonymous tracking on web
> Capture Snowplow web events without user or session identifiers, without IP addresses and without setting any cookies.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/anonymous-tracking/

The Snowplow web tracker allows you to track events anonymously. It enables anonymizing various user and session identifiers to support user privacy in case consent for tracking the identifiers is not given. This means that no user identifiers are sent to the Snowplow event collector. By default, anonymous tracking is not enabled.

Anonymous tracking can be configured on initialization, but can be reset later. You may wish to toggle this functionality on or off during a page visit, for example when a user accepts a cookie banner you may not want to disable anonymous tracking, or when a user logs in to your site.

For information about tracking user consent interactions and GDPR basis for processing, check out [this page](/docs/sources/web-trackers/tracking-events/consent-gdpr/). Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

On web, the following [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) can be anonymized:

- Client-side user identifiers:

  - `user_id`, a business identifier provided by you
  - `domain_userid`, set automatically by the tracker from the cookies
  - `userId` in the [session](/docs/sources/web-trackers/tracking-events/session/) entity, set automatically by the tracker to the same value as `domain_userid`
  - `tabId` in the [browser](/docs/sources/web-trackers/tracking-events/browsers/) entity, set automatically by the tracker

- Client-side session identifiers:

  - `domain_sessionid` and `domain_sessionidx`, set automatically by the tracker from the cookies
  - `previousSessionId` in the session entity, set automatically by the tracker

- Server-side user identifiers:
  - `network_userid` and `user_ipaddress`, set by the [Collector](/docs/pipeline/collector/)

## Configuring anonymous tracking

There are several levels to the anonymization depending on which of the three categories are affected. Set this using the [initialization configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### 1. Full client-side anonymization

```javascript
anonymousTracking: true,

stateStorageStrategy: 'cookieAndLocalStorage'
```

This mode will no longer track any user identifiers or session information. The tracker won't add the session entity to events, even if [session tracking](/docs/sources/web-trackers/tracking-events/session/) is enabled.

| Identifier          | Location in event | Included in event?  |
| ------------------- | ----------------- | ------------------- |
| `user_id`           | Atomic            | ❌                   |
| `domain_userid`     | Atomic            | ❌                   |
| `domain_sessionid`  | Atomic            | ❌                   |
| `domain_sessionidx` | Atomic            | ❌                   |
| `userId`            | Session entity    | ❌ no session entity |
| `sessionId`         | Session entity    | ❌ no session entity |
| `previousSessionId` | Session entity    | ❌ no session entity |
| `tabId`             | Browser entity    | ❌                   |
| `network_userid`    | Atomic            | ✅                   |
| `user_ipaddress`    | Atomic            | ✅                   |

Similar in behavior to setting `stateStorageStrategy: 'none'`, as it will store no values in cookies or localStorage, however by using `anonymousTracking` you can toggle this behavior on and off (useful for allowing events to be sent without user identifiers until cookie banners have been accepted).

Setting `stateStorageStrategy` to `cookieAndLocalStorage` or `localStorage` also allows for event buffering to continue working whilst not sending user information when `anonymousTracking` is enabled.

Anonymous tracking can be toggled on and off.

### 2. Client-side anonymization with session

```javascript
anonymousTracking: { withSessionTracking: true },

stateStorageStrategy: 'cookieAndLocalStorage'
```

This mode will continue to track session information but will track no user identifiers.

| Identifier          | Location in event | Included in event? |
| ------------------- | ----------------- | ------------------ |
| `user_id`           | Atomic            | ❌                  |
| `domain_userid`     | Atomic            | ❌                  |
| `domain_sessionid`  | Atomic            | ✅                  |
| `domain_sessionidx` | Atomic            | ✅                  |
| `userId`            | Session entity    | ❌ set to null UUID |
| `sessionId`         | Session entity    | ✅                  |
| `previousSessionId` | Session entity    | ❌                  |
| `tabId`             | Browser entity    | ❌                  |
| `network_userid`    | Atomic            | ✅                  |
| `user_ipaddress`    | Atomic            | ✅                  |

To achieve this, the tracker will use Cookies or local storage. For session tracking, `stateStorageStrategy` must be either `cookieAndLocalStorage` (default), `localStorage` or `cookie`. If this feature is enabled and the storage strategy is not appropriate, then full anonymous tracking will occur.

The Snowplow JavaScript Tracker performs sessionization client side. This allows anonymous session tracking to be done using client side storage without sending any user identifier fields to the collector.

### 3. Full anonymization/cookieless

```javascript
anonymousTracking: { withServerAnonymisation: true },

stateStorageStrategy: 'none',

eventMethod: 'post'
```

This mode will no longer track any user identifiers or session information, and will additionally prevent the Snowplow Collector from generating a `network_userid` cookie and capturing the users IP address. The same behavior described for above for Client side Anonymous tracking also applies.

| Identifier          | Location in event | Included in event?  |
| ------------------- | ----------------- | ------------------- |
| `user_id`           | Atomic            | ❌                   |
| `domain_userid`     | Atomic            | ❌                   |
| `domain_sessionid`  | Atomic            | ❌                   |
| `domain_sessionidx` | Atomic            | ❌                   |
| `userId`            | Session entity    | ❌ no session entity |
| `sessionId`         | Session entity    | ❌ no session entity |
| `previousSessionId` | Session entity    | ❌ no session entity |
| `tabId`             | Browser entity    | ❌                   |
| `network_userid`    | Atomic            | ❌                   |
| `user_ipaddress`    | Atomic            | ❌                   |

Setting `stateStorageStrategy` to `cookieAndLocalStorage` or `localStorage` also allows for event buffering to continue working whilst not sending user information when `anonymousTracking` is enabled. However for an experience that doesn't use any browser storage (cookieless), set `stateStorageStrategy` to `none`. This can be later toggled on, once a user accepts a cookie policy.

Server Anonymization requires the Snowplow Stream Collector v2.1.0+. Using a lower version will cause events to fail to send until Server Anonymization is disabled.

## Toggling anonymous tracking

You may wish to toggle this functionality on or off during a page visit, for example when a user accepts a cookie banner you may not want to disable anonymous tracking, or when a user logs in to your site.

### Enable anonymous tracking

> **Note:** Enabling Anonymous tracking will clear all current user, session and page data from events sent to the collector. Although not sent in requests to collector, existing user and session identifiers will not be removed from cookies or local storage. See below for information on how to clear user data.

If you wish to enable Anonymous Tracking, you can call:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking');
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';



enableAnonymousTracking();
```

***

which will enable client side anonymous tracking.

For full, cookieless anonymization, including anonymizing data within the Snowplow Collector (cookies and ip address), enable server anonymization too:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking', {

  options: { withServerAnonymisation: true }

});
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';



enableAnonymousTracking({

  options: { withServerAnonymisation: true }

});
```

***

Server Anonymization requires the Snowplow Stream Collector v2.1.0+. Using a lower version will cause events to fail to send until Server Anonymization is disabled.

If you want to enable anonymous tracking with session tracking, then you can use:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking', {

  options: { withSessionTracking: true }

});
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';



enableAnonymousTracking({

  options: { withSessionTracking: true }

});
```

***

From v3.1.0 it's also possible to change the `stateStorageStrategy` when enabling Anonymous Tracking, allowing you to switch off storage when turning anonymous tracking on:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking', { options: {}, stateStorageStrategy: 'none' }); // Available from v3.1.0
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';



enableAnonymousTracking({

  options: {},

  stateStorageStrategy: 'none'

});
```

***

### Disable anonymous tracking

To do this you can call the following methods:

**JavaScript (tag):**

```javascript
snowplow('disableAnonymousTracking');
```

or, if you wish to also adjust the `stateStorageStrategy` when enabling:

```javascript
snowplow('disableAnonymousTracking', {

  stateStorageStrategy: 'cookieAndLocalStorage'

});
```

**Browser (npm):**

```javascript
import { disableAnonymousTracking } from '@snowplow/browser-tracker';



disableAnonymousTracking();
```

or, if you wish to also adjust the `stateStorageStrategy` when enabling:

```javascript
import { disableAnonymousTracking } from '@snowplow/browser-tracker';



disableAnonymousTracking({

  stateStorageStrategy: 'cookieAndLocalStorage'

});
```

***

> **Note:** If configuring the tracker with `stateStorageStrategy: 'localStorage'` and anonymous tracking using `withSessionTracking: true`, then if you change to a `stateStorageStrategy` which prefer cookies such as `cookie` or `cookieAndLocalStorage` then the session identifiers will reset. To maintain session identifiers, ensure you use the same `stateStorageStrategy`.

### Clear user data

If you wish to clear all the cookies (except for the Collector `sp` cookie) and local storage values which contain user data when switching on anonymous tracking, or triggered by other actions on your site, you can call the following:

**JavaScript (tag):**

```javascript
snowplow('clearUserData');
```

**Browser (npm):**

```javascript
clearUserData();
```

***

From v3.1, this will also clear in memory session and user identifiers too. This ensures all possible identifiers are cleared and even if tracking is resumed you will see new session and user identifiers. If you'd like to preserve the in-memory session and user identifiers, for future events should you continue tracking after clearing the cookies, you can do so:

**JavaScript (tag):**

```javascript
snowplow('clearUserData', { preserveSession: true, preserveUser: true });
```

**Browser (npm):**

```javascript
clearUserData({ preserveSession: true, preserveUser: true });
```

***

## Respecting Do Not Track

Most browsers have a Do Not Track option which allows users to express a preference not to be tracked. You can respect that preference by setting the `respectDoNotTrack` field of the [initialization configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) to `true`. This prevents cookies from being sent and events from being fired.

## Opt-out cookie

Similar in function to Do Not Track, it's possible to set an [opt-out cookie](/docs/sources/web-trackers/cookies-and-local-storage/#opt-out-cookie) to prevent all events being tracked.

---

# Configure how events are sent by the web trackers
> Configure request methods, payload sizes, retries, and custom headers for event transmission to the collector.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/configuring-how-events-sent/

It's possible to add a fine-grained configuration for exactly how the web tracker should track and send events.

## Configuring the request

### Base64 encoding

Context entities and custom self-describing events can be encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the `encodeBase64` field of the [tracker configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

This setting is enabled by default for GET requests. For POST requests, it is disabled by default as it doesn't provide the same benefits and can result in larger payload size and is more difficult to debug.

### Number of events per request

The default `bufferSize` is 1, i.e. an event will be processed into a single request and sent as soon as it is tracked. This can be increased to send events in batches, using the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

The `bufferSize` property is only relevant when making POST requests, [see below](#post-support).

### Maximum payload size

**POST requests**

Because the Snowplow Stream Collector can have a maximum request size, the Tracker limits POST requests to 40000 bytes. If the combined size of the events in `localStorage` is greater than this limit, they will be split into multiple POST requests. You can override this default using a `maxPostBytes` in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

**GET requests**

By default, there is no limit on the maximum size of GET requests – the tracker will add to queue and try to emit all GET requests irrespective of their size. However (since version 3.4), there is an optional `maxGetBytes` parameter which serves two purposes:

1. It prevents requests over the threshold in bytes to be added to event queue and retried in case sending them is not successful.
2. It sends events over the threshold as individual POST requests (same as for `maxPostBytes`).

The size of GET requests is calculated for their full GET request URL.

**Collector limit**

The Snowplow Stream Collector cannot process requests bigger than 1MB because that is the maximum size of a Kinesis record.

### Custom request headers

From v3.2.0, you are able to set custom headers with an `eventMethod: "post"` and `eventMethod: "get"` (Except for IE9). This functionality should only be used in the case where a Proxy or other Collector type is being used which allows for custom headers to be set on the request.

> **Warning:** Adding additional headers without returning the appropriate CORS Headers on the OPTIONS request will cause events to fail to send.

```javascript
customHeaders: {

  'Content-Language': 'de-DE, en-CA',

}
```

Set this in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### Disabling sending credentials with requests

The `credentials` option in the tracker configuration controls whether cookies are sent with requests to a Snowplow Collector running on a different domain. The default setting is `include` to always include the cookies. The setting translates to the `credentials` option in the fetch API, see [the documentation here](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#including_credentials).

```json
credentials: 'omit'
```

Before v4, this configuration was provided using the `withCredentials` flag.

### Sent timestamp for GET events

By default, the tracker sends POST requests. If you update the `eventMethod` to `get`, the tracker will send events using GET requests. These events will contain a `stm` [timestamp parameter](/docs/events/timestamps/) recording when the event was sent.

If you need to make requests as small as possible, you can omit the `stm` parameter by setting the `useStm` [configuration option](/docs/sources/web-trackers/tracker-setup/initialization-options/) to `false`. This option was introduced in version 2.10.1. It's a niche parameter that you should only use if you have a specific reason to do so.

POST requests always include the sent timestamp.

## Network protocol and method

### Setting the request protocol

Normally the protocol (http or https) used by the Tracker to send events to a collector is the same as the protocol of the current page. You can force the tracker to use https by prefixing the collector endpoint with the protocol. For example:

**JavaScript (tag):**

```javascript
newTracker('sp', 'https://{{collector_url_here}}', {

  appId: 'my-app-id'

}
```

**Browser (npm):**

```javascript
newTracker('sp', 'https://{{collector_url_here}}', {

  appId: 'my-app-id'

}
```

***

### GET support

By default, events are sent by GET. This can be changed using the `eventMethod` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### POST support

If you set the `eventMethod` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) to `post`, the tracker will send events using POST requests rather than GET requests. In browsers which do not support cross-origin XMLHttpRequests (e.g. IE9), the tracker will fall back to using GET.

`eventMethod` defaults to `post`. `get` can be used for GET requests.

The main advantage of POST requests is that they circumvent Internet Explorer’s maximum URL length of 2083 characters by storing the event data in the body of the request rather than the querystring.

You can also batch events sent by POST by setting a numeric `bufferSize` field in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This is the number of events to buffer before sending them all in a single POST. If the user navigates away from the page while the buffer is only partially full, the tracker will attempt to send all stored events immediately, but this often doesn’t happen before the page unloads. Normally the tracker will store unsent events in `localStorage`, meaning that unsent events will be resent when the user next visits a page on the same domain. The `bufferSize` defaults to 1, meaning events are sent as soon as they are created.

We recommend leaving the `bufferSize` as the default value of 1. This ensure that events are sent as they are created, and reduces the chance of events being unsent and left in local storage, if a user closes their browser before a flush can occur (which happens on page visibility changing).

If you have set `bufferSize` to greater than 1, you can flush the buffer using the `flushBuffer` method:

**JavaScript (tag):**

```javascript
snowplow('flushBuffer');
```

**Browser (npm):**

```javascript
flushBuffer();
```

***

For instance, if you wish to send several events at once, you might make the API calls to create the events and store them and then and call `flushBuffer` afterwards to ensure they are all sent before the user leaves the page.

Note that if `localStorage` is inaccessible or you are not using it to store data, the buffer size will always be 1 to prevent losing events when the user leaves the page.

### `keepalive` option for Collector requests

The keepalive feature in the [fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) indicates that the request should be allowed to outlive the webpage that initiated it. It enables requests to the Snowplow Collector to complete even if the page is closed or navigated away from.

This option is disabled by default.

> **Note:** Browsers put a limit on keepalive requests of 64KB. You may control the size of the individual POST requests using the `maxPostBytes` option. But in case of multiple keepalive requests in parallel (may happen in case of multiple trackers), the limit is shared between them and may cause the requests to fail.

```ts
let tracker = newTracker("...", "...", {

    // ...

    keepalive: true

});
```

### Custom POST path

The POST path that is used to send POST requests to a collector can be changed with the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) value `postPath`.

`postPath` defaults to the standard path: `/com.snowplowanalytics.snowplow/tp2`

> **Warning:** While modifying `postPath` is generally recommended as a way to help avoid various ad-blocking tools when capturing events, it is also non-standard behavior that does not conform to the default Collector protocol.
>
> You must update the [Collector configuration](/docs/pipeline/collector/) to support the new path _before_ you send events to it with this setting. Otherwise, events will not be received by the Collector, or in some cases will be collected but will fail validation.
>
> Make sure that requests are supported by your Collector configuration or redirected to the Collector at the correct endpoint (normally this is `/com.snowplowanalytics.snowplow/tp2`).

## Retries

Unsuccessful requests are retried by default: the tracker retries on all 3xx, 4xx, and 5xx status codes except for 400, 401, 403, 410, and 422. Events in failed requests that are not retried are lost.

Starting with version 3.17 of the tracker, it is also possible to completely disable retry functionality, using the `retryFailedRequests` boolean option in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This option takes precedence over `retryStatusCodes` and `dontRetryStatusCodes` (see below).

### Connection timeout

When events are sent using POST or GET, they are given 5 seconds to complete by default. GET requests having a timeout is only available in 2.15.0.

`_connectionTimeout_: 5000`

This value is configurable when initialising the tracker ([configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/)) and is specified in milliseconds. The value specified here will effect both POST and GET requests.

**Warning:** Setting this value too low may prevent events from successfully sending to your collector or the tracker may retry to send events that have already arrived at the collector, as the tracker will assume the request failed on timeout, leading to duplicate events in the warehouse. **We recommend 5000 milliseconds as the minimum value and 10000 as the maximum value.**

### Custom retry HTTP codes

The tracker provides a retry functionality that sends the same events repeatedly in case GET or POST requests to the Collector fail. This may happen due to connection issues or a non-successful HTTP status code in Collector response.

Prior to version 3.5 of the tracker, requests receiving all 4xx and 5xx HTTP status codes in Collector response were retried. Since version 3.5, the behavior changed and became customizable:

By default, the tracker retries on all 3xx, 4xx, and 5xx status codes except for 400, 401, 403, 410, and 422. The set of status codes for which events should be retried or not is customizable. You can make use of the `retryStatusCodes` and `dontRetryStatusCodes` lists to specify them ([configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/)). Retry behavior can only be configured for non-successful status codes (i.e., >= 300).

```json
retryStatusCodes: [403], // override default behavior and retry on 403

dontRetryStatusCodes: [418] // force retry on 418
```

Please note that not retrying sending events to the Collector means that the events will be dropped when they fail to be sent. Take caution when choosing the `dontRetryStatusCodes`.

### Event queue behavior

The `useLocalStorage` [configuration option](/docs/sources/web-trackers/tracker-setup/initialization-options/) refers to event queue behavior. It was introduced in version 4.0.0. If you set it to `false`, events will be stored in memory only. If the page closes before events are sent, they'll be lost.

This is a **risky** setting that you should only use if you have a specific reason, such as compliance or privacy.

Previous versions of the tracker had a `useLocalStorage` option that referred to state storage. In version 3 [we deprecated this option](/docs/sources/web-trackers/migration-guides/v2-to-v3-migration-guide/) in favor of `stateStorageStrategy`. If you're upgrading from version 2 a